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PREFACE 


This  report  documents  the  results  of  a  human  factors  evaluation  of  the  Aerial  Gunner 
Scanner  Simulator  which  was  conducted  under  Work  Unit  1 123-B3-01,  Special 
Operations  Forces  (SOF)  Aircrew  Training  and  Mission  Preparation  Research.  The 
Laboratory  Work  Unit  Monitor  is  Dr.  Robert  T.  Nullmeyer,  Armstrong  Laboratory, 
Human  Resources  Directorate,  Aircrew  Training  Research  Division  (AL/HRA).  The 
work  was  supported  by  Work  Unit  1123-B2-06,  Aircrew  Training  Research  Support, 
under  Contract  F41624-95-C-5011.  The  Laboratory  Contractor  Monitor  is  Mr.  Daniel 
Mudd;  the  Laboratory  Technical  Monitor  is  Dr.  Robert  T.  Nullmeyer. 
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human  factors  evaluation 

OF  THE  AERIAL  GUNNER  SCANNER  SIMULATOR 
INTRODUCTION 

This  report  describes  a  human  factors  evaluation  of  the  Aerial  Gunner  Scanner  Simulator 
(AGSS)  that  was  installed  at  Kirtland  Air  Force  Base  (KAFB),  New  Mexico  and I  ts  being  operated 
by  the  58  Special  Operations  Wing  (SOW).  The  evaluation  was  performed  durrng  the  period 
Januarv-Aueust  1996,  with  survey  data  collected  in  March  and  Apnl  of  the  same  year.  The  effo 
as  pari  of  a  cooperative  research  effort  between  the  58  Training  Support  Squadron 
(TRSS)  and  the  United  States  Air  Force  (USAF)  Armstrong  Laboratory,  Human  Resources 
Directorate,  Aircrew  Training  Research  Division  (AL/HRA). 

Declared  ready  for  training  (RFT)  in  March  1996,  the  AGSS  was  develoPedto  provide  a 
low  cost  high  fidelity  virtual  environment  for  training  MH-53J  Pave  Low  and  M/HH-60G  Pave 
Hawk  flight  engineers  (FEs)  and  aerial  gunner/scanners  (AG/Ss).  The  AGSS  is  a  state-of-the-art 
reconfigurable  simulator  that  can  train  up  to  three  crewmembers  at  a  time.  A  dynamic  virbia^ 
environment  is  presented  through  a  head-tracked,  full-color,  helmet-mounted  display  (HMD) 
system  that  can  support  both  daytime  and  night  vision  goggle  (NVG)-mght  simulation.  Using  a 
three  degree-of-freedom  motion  system,  aural  cueing,  a  powerful  seven-channel  image  generator 
(IG)  and  digital  computation  system,  the  AGSS  is  capable  of  simulating  air,  sea,  an  terrain 
scenes;  ground  and  airborne  targets;  and  weapons  effects,  including  tracer  path  and  bullet  impact 

(Reed,  1996). 

Because  the  AGSS  represents  the  first  use  of  virtual  reality  (VR)  for  aircrew  training  at  the 
58  SOW  an  assessment  of  the  human  factors  aspects  of  the  system  was  requested  by  the  58  TRSS. 
As  one  of  the  USAF  Air  Education  and  Training  Command  (AETC)  training  centers  for 
excellence,  the  58  SOW  deemed  it  necessary  to  conduct  a  thorough  evaluation  of  the  AGSS  from  a 
human  factors  standpoint  before  incorporating  the  system  into  its  impressive  cadre  of  advance 

weapon  system  trainers  (WSTs). 

The  evaluation  was  designed  to  achieve  three  major  objectives.  First,  the  information 
collected  during  the  evaluation  was  rapidly  disseminated  to  key  personnel  within  the  58  TRSS  to 
ensure  that  high  payoff,  incremental  design  improvements  to  the  AGSS  could  be  made  as  soon  as 
possible.  Second,  the  effort  sought  to  identify  the  skills  most  affected  by  use  of  the  A 
thereby  laying  the  foundations  for  a  subsequent  assessment  of  the  effectiveness  of  the  AGSS  as  a 
training  device  Third,  the  research  was  designed  to  collect  “lessons  learned”  information  an 
other  insights  from  subject-matter  experts  (SMEs)  that  could  be  used  in  the  design,  development, 
and  implementation  of  other  VR-based  systems. 

The  remainder  of  the  report  is  organized  into  four  sections.  We  begin  by  providing  some 
background  on  the  AGSS.  This  includes  a  description  of  the  training  and  design  requirements  tor 
the  system,  a  discussion  of  how  VR  technology  is  being  applied  to  training  and  mission  rehearsal 
(MR)  a  consideration  of  applying  human  factors  analyses  to  VR  technology,  and  a  delineation  of 
the  scope  of  the  human  factors  survey  that  we  performed.  In  the  next  section  we  describe  the 
methods  used  to  conduct  the  survey,  including  a  description  of  the  AGSS,  the  subjects,  the  survey 
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instrument,  data  collection  procedures,  and  the  rationale  behind  the  statistical  methods  used.  The 
results  section  presents  the  overall  findings  of  the  survey,  followed  by  separate  analyses  of  the 
ratings  and  qualitative  data.  Finally,  we  summarize  the  major  findings,  offer  some 
recommendations  for  AGSS  design  and  use,  and  briefly  introduce  our  research  on  AGSS  training 
effectiveness  and  user  acceptance. 

Evolution  of  Training  and  Design  Requirements  for  the  AGSS 

Since  1990,  the  58  SOW  has  been  vigorously  developing  a  sophisticated  set  of  simulation 
and  electronic  training  technologies,  as  the  USAF  undergoes  massive  changes  to  become  combat 
mission  ready  for  the  next  century.  Pursuing  a  path  of  innovative  acquisition  and  an  aggressive 
strategy  for  system  development,  along  with  an  integrated  approach  toward  academic  instruction, 
flightline  training,  and  simulation,  the  58  SOW  has  acquired  a  “can-do”  reputation  for  fielding 
sophisticated,  reliable  simulators  in  a  short  time  period  and  incorporating  these  technologies  into 
ongoing  Mission  Qualification  (MQ)  and  combat  mission  training  programs  with  minimal 
disruptions.  As  of  1996,  the  impressive  capabilities  available  at  the  58  SOW  included  eight  high 
fidelity  networked  flight  simulators;  electronic  classrooms  for  advanced  academic  instruction;  a 
comprehensive  database  generation  facility  to  construct  photo-enhanced,  geospecific  visual 
databases  for  any  location  in  the  world;  an  integrated  training  and  mission  support  system, 
employing  computer-aided  instruction  (CAI);  a  cadre  of  database  modelers  and  programmers; 
various  hardware  and  software  assets  to  support  construction  of  photo-enhanced,  geospecific 
databases;  and  functional  links  to  multiple  nodes  on  the  Department  of  Defense’s  (DOD) 
Distributed  Interactive  Simulation  (DIS)  constructive  battlefield  simulation  network. 

As  recently  as  1990,  the  flight  simulators  were  only  beginning  to  come  on  line  at  KAFB. 
The  first  of  these  was  the  MH-53J  WST — and  its  associated  database  generation  system  (DBGS) 
and  mission  planning  system.  The  USAF  wanted  to  assess  the  suitability  of  the  MH-53J  WST  for 
supporting  short  suspense  MR  and  training  activities.  An  initial  operational  effectiveness 
evaluation  was  conducted  in  1991  in  the  context  of  a  joint  Air  Force-Army  training  exercise  at 
White  Sands,  New  Mexico  (Nullmeyer,  Bruce,  Conquest,  &  Reed,  1992).  Simulating  a  strike 
mission  to  recover  critical  equipment  from  a  hostile  third-world  nation,  the  concept  of  operation 
called  for  five  helicopter  crews  (3  MH-53Js,  2  MH-60Gs)  to  insert  a  90-person  Army  team  (assault 
and  support)  into  the  objective  area  during  two  consecutive  nights.  Throughout  the  assessment, 
AL/HRA  scientists  provided  behavioral  research  support  by  directly  observing  the  mission 
preparation  process,  interviewing  selected  mission  participants,  and  administering  formal,  open- 
ended  questionnaires  following  both  MR  and  mission  execution  (Nullmeyer,  Bruce,  &  Spiker, 


The  overall  assessment  of  the  MH-53J  WST  was  positive,  as  evidenced  by  the  high  marks 
given  the  system  by  all  the  MH-53J  and  MH-60G  pilots  and  FEs  who  were  queried.  Indeed,  crew 
confidence  in  their  mission  plan  was  increased  as  a  function  of  rehearsal  in  the  WST  (Nullmever 
Spiker,  &  Reed,  1996).  v  y  ’ 

Although  the  aircrews  strongly  endorsed  the  MR  capability  of  the  system,  critiques  by  the 
AG/Ss  contained  some  key  points.  Since  the  WST  only  simulates  the  cockpit  for  the  two  pilots 
and  one  FE,  the  AG/Ss  did  not  rehearse  in  the  device.  The  AG/Ss  suggested  that  the  entire  crew 


2 


be  included  in  the  MR.  They  also  indicated  that  the  scanners  need  a  visual  capability  of  their  own, 
and  cited  the  need  to  communicate  with  the  cockpit  crew  during  the  rehearsal  to  coordinate  tactical 
activities  and  maintain  situation  awareness. 

Based  on  this  feedback,  it  was  recommended  that  a  simulation  capability  be  developed  that 
allows  the  AG/Ss  to  view  the  visual  terrain  database  just  as  the  pilots  and  FEs  do,  only  from  then- 
vantage  point  at  the  rear  and  sides  of  the  aircraft  (Nullmeyer,  et  al.,  1992).  Not  only  would  such  a 
capability  permit  the  training  of  crew-level  tasks,  as  the  entire  crew  would  be  present  in  the  WST, 
it  would  aid  development  of  key  gunnery  and  scanning  skills  while  reducing  demands  on  the 
flightline  to  schedule  sorties  solely  to  accomplish  AG/S-unique  mission  training  events. 

In  parallel  with  the  desire  for  full  crew  training,  the  58  SOW  explored  developing  a 
gunnery  simulator  to  serve  as  an  electronic  part-task  trainer  (Reed,  1996).  While  developing  the 
requirements  for  an  AG/S  training  simulator,  the  58  TRSS  explored  promising  new  technologies 
for  incorporation  into  their  formal  specification.  These  included  low-cost,  small  motion  platforms; 
ruggedized,  full-color  HMDs;  more  accurate  head-tracking  systems;  and  smaller  high  fidelity  IGs. 

In  May  1993,  a  contract  was  awarded  to  design  the  AGSS.  At  the  outset,  the  AGSS  was 
expected  to  provide  full  crew  training  and  MR.  Not  only  would  the  AGSS  provide  the  first  true 
MR  capability  for  enlisted  crewmembers,  it  would  do  so  cost  effectively  by  allowing  squadron 
commanders  to  trade  high  value  training  flight  hours  for  simulator  time  (Reed,  1996). 

During  the  design  phase,  four  key  factors  would  alter  the  physical  configuration  of  the 
device  as  well  as  shape  the  corresponding  human  factors  evaluation.  First,  by  using  a  modular 
architecture,  the  AGSS  was  to  be  reconfigurable  to  support  either  MH-53J  or  MH-60G  training. 
This  flexibility  would  not  only  increase  the  number  and  type  of  crewmembers  to  be  trained,  it  also 
expanded  the  range  of  training  events  to  be  covered. 

Second,  the  small  size  of  the  simulator  device  necessitated  that  the  AGSS  instructor/ 
operator  station  (IOS)  not  be  mounted  on  the  motion  platform,  but  instead,  be  placed  on  the 
facility  floor  outside  the  MH-53J  WST.  This  decision  affects — and  potentially  limits — the  ways 
that  instructors  may  interact  with  their  trainees.  The  resulting  effects  on  both  instructor  and  trainee 
must  be  captured  in  the  human  factors  evaluation. 

Third,  to  accommodate  the  heavy  training  schedule  of  the  simulators  at  the  58  SOW,  the 
AGSS  needed  to  be  designed  for  operation  in  both  an  independent  (stand-alone)  mode  and 
integrated  mode  (networked  with  a  cockpit  simulator),  enabling  whole  crew  operations.  The 
different  requirements  for  operation,  and  the  added  complexities  of  using  the  AGSS  while 
networked  with  other  WSTs,  must  be  reflected  in  a  comprehensive  human  factors  assessment. 

Fourth,  revolutionary  advances  in  IG  capability  during  the  design  phase  created  the 
exciting  possibility  of  incorporating  full  VR  into  routine  training.  By  scanning  digital 
representations  of  the  aircraft  interior  and  displaying  those  on  the  AG/S’s  HMD,  coupled  with 
position-referenced  aural  cues  and  vibratory  feedback  from  the  guns,  the  AGSS  has  the  potential 
to  offer  a  truly  immersive  training  environment.  These  novel  aspects  must  also  be  explored  in  a 
human  factors  analysis  of  the  AGSS.  However,  before  further  discussion  of  the  general 
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methodological  requirements  of  a  comprehensive  human  actors  evaluation,  in  the  next 
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VR  Application  Issues 


From  both  a  technological  and  societal  perspective,  VR  has  seen  an  unprecedented 
explosion  in  popularity,  interest,  and  debate  over  the  last  few  years.  While  multiple  definitions 
abound,  a  common  conception  of  VR  is  a  type  of  interactive  simulation  that  includes  the  user  as  a 
necessary  component.  Recently,  however,  it  is  has  been  noted  (e.g.,  Thurman  &  Mattoon,  1991) 
that  such  conceptions  are  overly  restrictive,  since  VR  also  encompasses  the  experience  the  user 
has  m  the  artificial  environment.  Within  DOD,  the  prevailing  conception  of  VR  is  as  an 
Interactive,  usually  computer-mediated,  experience  characterized  by  a  suspension  of  (or 
inattention  to)  disbelief  in  the  unreality  of  the  experience”  (DOD,  1 994,  p.  1 8). 


In  attempting  to  analyze  the  potential  uses  of  VR  in  simulating  a  combat  training 
environment,  it  should  be  noted  that  DOD  Directive  5000.59  defines  simulation  as  a  “method  for 
implementing  a  model  over  time  (p.  4).”  In  turn,  a  model  is  viewed  as  a  “physical,  matWnatiVai 
or  otherwise  logical  representation  of  a  system,  entity,  phenomenon,  or  process  (p.  4).” 
Accordingly,  when  assessing  the  effectiveness  of  a  VR-based  training  device,  it  is  important  to 
address  how  accurately  the  simulation  represents  the  objects  of  interest,  how  realistically  the 
modeled  objects  behave  over  time,  and  how  efficiently  the  modeled  environment  is  delivered  to 
the  user. 


Given  the  user’s  large  role  in  the  VR  experience,  it  stands  to  reason  that  an  assessment  of 
the  human  fectors  aspects  of  the  implementation  is  vital,  as  it  will  clearly  influence  how  effective 
the  simulation  will  be.  In  this  regard,  it  is  important  to  keep  in  mind  that  VR  technology  is  not  a 
singte  technology  or  medium,  such  as  an  HMD  or  a  DataGlove™.  Rather,  it  is  an  integrated 
human-computer  system  (Thurman  &  Mattoon,  1991).  Consequently,  when  VR  is  incorporated 
into  a  training  device,  all  aspects  of  the  human-computer  system— including  input,  interface, 
processing,  and  output — must  be  evaluated,  both  separately  and  in  combination. 

. _  Among  the  various  concepts  used  to  explain  VR,  three  are  essential:  virtual  environment, 

immersion,  and  presence.  A  virtual  environment  is  a  simulated  space  in  which  a  viewer  or  user 
may  interact.  Typically,  a  physical  simulation  of  a  vehicle,  such  as  an  aircraft  or  a  tanV  is  the 
interface  between  the  individual  in  the  simulation  and  the  virtual  environment.  For  an  individual 

to  interact  directly  in  a  virtual  environment,  some  or  all  of  the  following  conditions  must  be  met 
(Knerr,  et  al,  1993): 

•  free  motion  of  the  eyepoint  in  space; 

•  three-dimensional  real-time  interactive  graphics,  with  stereopsis  as  needed; 

•  multiple  senses  beyond  visual  (e.g.,  audition,  touch); 

•  direct  manipulation  of  objects;  and 

•  multiple  interacting,  mutually  visible  humans. 
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Considering  these  criteria,  the  AGSS  and  its  associated  IG-generated  visual  database  certainly 
qualifies  as  providing  a  virtual  environment  for  AG/S  trainees. 

Within  the  VR  domain,  immersion  is  achieved  when  one  or  more  of  the  user’s  senses 
(typically  vision  and  audition)  are  isolated  from  the  surrounding  environment  and  fed  only 
information  from  a  computer  simulation  (Pimentel  &  Teixeira,  1993).  An  HMD  or  large-dome 
display  is  typically  required  to  achieve  such  immersion  However,  true  immersion  is  said  to  occur 
when  the  user  interacts  with  the  virtual  environment  as  if  it  were  his/her  actual  physical 
environment.  By  this  definition,  it  would  be  difficult  for  an  outside  observer  to  attract  the 
attention  of  a  totally  “immersed”  user.  This  situation  might  pose  some  potential  drawbacks  in  a 
training  setting,  where  the  instructor  requires  periodic,  real-time  contact  with  the  trainee. 
Interestingly,  there  appear  to  be  individual  differences  among  users  in  their  susceptibility  to 
immersion,  much  as  there  is  to  hypnosis  (Knerr,  et  aL,  1993). 

A  related  VR  concept  is  presence,  which  is  the  subjective  experience  of  being  in  one  place 
when  one  is  physically  in  another  (Bailey  &  Witmer,  1994).  In  a  sense,  presence  can  be  viewed  as 
a  weaker  form  of  immersion,  since  the  user’s  senses  do  not  need  to  be  fully  engaged. 
Importantly,  this  may  be  a  more  realistic  and  practical  goal  for  a  VR-based  training  technology. 
In  this  regard,  there  is  evidence  to  suggest  that  users  can  still  experience  presence  while  modifying 
the  viewpoint  in  the  virtual  environment,  thus  improving  their  overall  spatial  knowledge  of  objects 
in  the  virtual  world  (Mowafy  &  Miller,  1993). 

As  a  technology,  there  has  been  a  rapid  expansion  of  VR  capabilities  since  the  early  1980s. 
Depending  on  price  and  one’s  task  area  of  interest,  it  is  possible  to  distinguish  among  four  levels 
of  VR  (DOD,  1994).  Entry  level  VR  includes  a  cathode  ray  tube  (CRT)  graphic  display;  desk  top 
computer;  and  a  two-dimensional  input  device,  such  as  a  mouse,  trackball,  or  joystick.  Expansion 
into  a  Level  2  or  basic  VR  adds  some  basic  interaction  and  display  enhancements.  These  might 
include  a  stereographic  viewer  and  a  more  capable  input  device.  The  latter  might  be  either  a 
DataGlove™  or  a  multidimensional  (3-  or  6-dimension)  mouse  or  joystick. 

Stepping  up  to  Level  3  or  advanced  VR  entails  the  addition  of  a  rendering  accelerator  card 
for  fester  display  of  animated  graphics  and  possibly  a  sound  card  for  true  3-D  audio  output.  As 
parallel  processors  have  come  down  in  price,  it  has  become  more  feasible  to  render  real-time 
imaging  video  over  a  larger  display  area.  Such  a  capability  adds  realism  to  the  visual  presentation 
and  typically  enhances  the  user’s  interest  and  enjoyment  of  the  experience.  Moreover,  including 
audio  feedback  has  been  shown  to  dramatically  enrich  the  user’s  VR  experience  (Miner,  1994). 

One  reaches  the  highest  level  of  VR  (Level  4)  by  adding  some  type  of  immersive  display 
system.  At  the  present  time,  the  choices  are  HMDs,  multiple  large  project  displays,  or  binocular 
omni-orientation  monitors  (BOOMs).  A  BOOM  device  has  the  viewer  look  through  a 
stereoscopic  display  whose  field  of  regard  is  swiveled  around  a  fixed  boom  platform.  In  each 
case,  it  is  typically  assumed  that  the  user  will  have  an  “immersive  experience”  if  the  display  can 
provide  at  least  60°  of  horizontal  field  of  view  (FOV).  An  immersive  system  might  also  involve 
some  form  of  haptic  interaction  mechanism  so  that  the  user  can  feel  virtual  objects. 
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Given  the  sophisticated  computational  and  display  capabilities  associated  with  the  AGSS, 
this  system  clearly  satisfies  all  the  criteria  for  a  Level  4,  the  highest  level  of  immersion  for  VR 
technology.  While  achieving  immersion  is  a  worthy  goal  from  a  technological  standpoint,  it 
remains  to  be  seen  whether  that  sophistication  translates  into  improved  user  performance  or 
increased  training  effectiveness. 

Although  there  is  now  considerable  commercial  interest  in  VR,  its  application  to  combat 
mission  training  is  still  in  its  infancy.  For  example,  the  Army  Research  Institute  (ARI)  is  exploring 
the  use  of  VR  technology  to  create  virtual  environments  for  training  spatial  knowledge  with 
dismounted  soldiers  (Bailey  &  Witmer,  1994).  Researchers  are  finding  evidence  that  participants 
can  learn  to  navigate  through  real-world  places  by  training  in  a  virtual  environment,  although  such 
training  is  not  as  effective  as  walking  through  an  actual  building.  On  the  other  hand,  the 
differences  between  VR  and  real-world  navigation  training  in  producing  accurate  spatial 
knowledge  may  dissipate  with  greater  amounts  of  practice. 

The  Defense  Advanced  Research  Projects  Agency  (DARPA)  and  the  Army  Test  and 
Evaluation  Command  (TECOM)  are  conducting  a  joint  project  to  determine  the  feasibility  of 
using  VR  technology  to  train  automobile  drivers  to  achieve  high  levels  of  performance  on  an 
instrumented  test  track  (Kuhl  &  Wargo,  1994).  Initial  transfer  of  training  results  are  encouraging, 
and  through  use  of  virtual  prototyping  simulation,  the  investigators  are  exploring  the  applicability 
of  VR  to  conduct  tactical  battlefield  training  in  DIS-based  exercises. 

Not  surprisingly,  the  USAF  is  sponsoring  research  to  study  the  effectiveness  of  VR 
technology  in  training  situation  awareness  among  fighter  pilots  (Hettinger,  Nelson,  &  Hass,  1994; 
Mowafy  &  Congdon,  1994).  Researchers  at  AL/HRA  are  collecting  data  to  identify  those  spatial 
knowledge  tasks  whose  performance  appears  to  be  aided  by  training  in  a  virtual  world.  Some  of 
the  areas  where  VR  technology  appears  to  hold  promise  involve  providing  pilots  with  multiple 
perspectives  (i.e.,  inside  the  cockpit  vs  an  outside  or  exocentric  point  of  view)  during  an 
engagement,  navigating  through  the  environment  in  three-dimensions  to  estimate  energy 
management  performance,  simultaneous  comparison  of  flight  trajectories  flown  by  students  and 
experts,  and  joint  use  of  icons  and  audio  to  give  students  immediate  feedback  concerning 
performance  in  a  simulated  mission  (Mowafy  &  Congdon,  1994). 

While  the  potential  for  VR  technology  to  positively  impact  combat  mission  training  is 
quite  real,  there  are  nevertheless  some  problem  areas  that  have  been  highlighted  in  past 
implementations.  These  problem  areas  must  be  resolved  in  any  successful  application  and, 
accordingly,  will  be  assessed  in  our  human  factors  survey. 

There  have  been  numerous  reports,  mostly  anecdotal,  of  problems  experienced  by  HMD 
wearers  in  simulation  settings.  The  most  commonly  reported  problems  include  eye  strain,  blurred 
vision,  headaches,  and  occasionally,  debilitating  “‘flash  backs”  that  occur  several  hours  after  the 
simulation  experience.  In  some  cases,  inadequate  performance  of  the  head-tracking  mechanism 
causes  significant  transport  delays,  producing  system  lags  that  result  in  scene  blurring  as  the  user 
moves  his/her  gaze  across  the  display.  Other  perceptual  problems  have  been  reported  due  to  the 
insufficient  display  resolution  in  many  present-day  HMD  image  sources  (e.g.,  liquid  crystal 
displays  or  LCDs)  for  rendering  detailed  graphic  scenes  (DOD,  1994). 
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Another  class  of  problems  with  VR  technology  entails  user  performance.  For  example, 
there  are  frequent  reports  of  users  becoming  disoriented  while  navigating  in  virtual  environments 
(e.g.,  Mowafy  &  Congdon,  1994).  Since  the  user’s  bearing  is  variable  in  a  virtual  world,  it  is 
often  difficult  to  maintain  a  consistent  azimuthal  relationship  between  one’s  self  and  significant 
objects  (e.g.,  targets)  in  the  environment.  Consistent  with  this  problem,  there  are  many  reports  of 
users  walking  into  objects,  both  virtual  and  real  world,  while  navigating  through  VR  environments 
(DOD,  1994). 

Perhaps  the  largest  potential  problem  concerns  the  apparently  high  incidence  of  simulator 
sickness  observed  in  a  number  of  previous  VR  implementations.  As  in  any  simulation  setting, 
discrepancies  between  vection  (perceived  self-motion)  and  visual  feedback  from  the  VR 
simulation  can  cause  discomfort,  a  feeling  of  nausea,  or  outright  sickness  among  many  subjects 
(Lane,  Kennedy,  &  Jones,  1994).  The  percentage  of  VR  users  who  experience  some  degree  of 
sickness  varies,  with  one  study  (Bailey  &  Witmer,  1994)  excusing  only  1 6%  of  their  subjects  due 
to  illness,  while  Knerr  et  aL  (1994)  reported  simulator  sickness  symptoms  in  most  of  their  study 
participants. 

The  true  incidence  and  extent  of  simulator  sickness  will  no  doubt  vary  as  a  function  of 
many  conditions.  For  example,  there  is  some  evidence  to  suggest  that  the  amount  of  presence 
experienced  in  the  VR  setting  is  negatively  related  to  the  severity  of  simulator  sickness  reported 
for  that  environment  (Knerr  et  aL,  1994).  Moreover,  there  are  also  data  suggesting  that  simulator 
sickness  may  adversely  affect  performance,  even  for  those  participants  who  do  not  experience 
severe  symptoms  (Bailey  &  Witmer,  1994).  Anecdotally,  some  believe  that  experiencing  a  VR 
simulation  while  on  a  motion  base  (as  opposed  to  a  fixed-base)  will  alleviate  many  simulator 
sickness  problems  (DOD,  1994).  Whatever  the  mechanism(s),  it  is  clear  that  definitive  steps  will 
need  to  be  taken  to  reduce  the  incidence  of  simulator-induced  illness  before  VR  technology  can  be 
implemented  as  a  training  medium  on  a  large  scale. 

Viewed  in  the  long  term,  it  is  reasonable  to  expect  that  VR  will  prove  effective  for  some 
training  tasks  and  not  for  others.  As  such,  VR  will  not  be  the  “silver  bullet”  that  revolutionizes 
the  military’s  approach  to  training  nor  radically  alters  how  training  is  conducted  at  the  squadron 
level  Consequently,  the  job  of  the  behavioral  scientist  is  to  identify  the  characteristics  of  the 
user’s  tasks  whose  performance  are  aided,  hindered,  or  unaffected  by  VR  so  that  the  latter  may  be 
judiciously  and  cost-effectively  applied  to  other  weapon  training  systems  (Lane  et  aL,  1994). 
Such  identification  is  the  guiding  focus  of  the  human  factors  evaluation  described  in  this  report. 

Human  Factors  Analysis  of  VR  Technology 

Human  factors  is  a  subdiscipline  of  psychology  whose  primary  objective  is  to  design 
machines  and/or  work  environments  that  accommodate  the  limits  of  the  human  user  (Wickens, 
1984).  As  typically  practiced,  a  human  factors  analysis  will  conceptualize  the  human-machine 
environment  as  a  total  system  whose  components  and  component  relationships  may  then  be 
examined  in  more  detaiL  Figure  1  depicts  the  user-machine  environment  as  typically  conceived  in 
a  human  factors  analysis. 
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Figure  1.  Typical  Human  Factors  Conception  of  the  User-Machine  Relationship. 

As  seen  in  the  figure,  the  limitations  and  capabilities  of  the  human  user  can  be  studied  at 
several  points  in  the  system.  The  user’s  abilities  to  make  inputs  to  the  machine  should  be 
considered.  Depending  on  the  application,  the  user  might  make  inputs  to  the  machine  using 
components  such  as  a  keyboard,  joystick,  mouse,  touch  screen,  or  voice  recognition  system. 
Each  of  these  in  turn  might  involve  interaction  from  the  user  using  various  parts  of  the  body,  such 
as  fingers,  arms,  and  vocal  chords.  The  corresponding  human  capacities  in  such  areas  as  strength, 
dexterity,  speed,  or  stamina  need  to  be  reviewed  to  determine  their  suitability  for  the  machine’s 
control  mechanisms  (Van  Cott  &  Warrick,  1972). 

As  machine  inputs  are  processed,  their  results  are  displayed  back  to  the  user  in  some  form. 
Display  choices  might  be  a  CRT,  HMD,  large  dome,  and  voice,  to  name  just  a  few.  Again,  human 
limitations  in  the  sensory  domain  of  the  display  (vision,  audition,  touch)  must  be  considered  to 
ensure  that  the  displayed  information  is  sufficiently  large,  detectable,  perceptible,  legible,  and  so 
forth  (Grether  &  Baker,  1972). 

The  circles  in  the  middle  of  the  figure  encompass  what  is  rapidly  becoming  the  most 
critical  area  of  human  factors — information  processing.  Thus,  regardless  of  the  method  of  display 
or  the  medium  of  control,  substantial  processing  will  occur  by  both  the  human-user  and  the 
machine.  Since  the  processing  of  both  entities  is  hidden  from  direct  observation,  the  challenges 
feeing  the  human  fectors  analyst  are  greatest  here  (Kantowitz  &  Sorkin,  1983).  On  the  machine 
side,  the  focus  will  be  to  ensure  that  the  system  responds  sufficiently  soon  after  input  so  that  the 
user  does  not  experience  any  lag  or  delay  that  might  degrade  his/her  performance.  Responding  to 
the  user  side,  the  goal  is  to  design  the  system  so  that  user’s  cognitive  processing  capacities 
(attention,  short-term  memory,  long-term  memory,  etc.)  are  not  overwhelmed. 

With  regard  to  understanding  the  inpact  of  VR  technology  on  these  fectors,  thorough 
human  fectors  analyses  are  essential  when  new  VR  devices  are  introduced  into  training  programs, 
because  limitations  in  the  user-device  interactions  will  undoubtedly  establish  upper  limits  on  the 
device’s  training  effectiveness.  There  are  several  key  elements  of  a  well-designed  human  fectors 
study.  First,  the  analysis  should  attempt  to  distinguish  between  problems  resulting  from 
limitations  in  the  user’s  basic  sensory-cognitive-motor  capacities  versus  those  arising  from 
deficiencies  in  the  machine.  Such  a  distinction  is  necessary  so  that  when  problems  in  device  use 
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are  encountered,  the  proper  solution(s)  can  be  determined — modifying  the  device,  providing 
additional  training  to  the  user,  or  both. 

As  applied  to  the  AGSS  as  a  VR  device,  the  human  factors  analysis  looks  at  the  users’ 
experiences  of  immersion,  presence,  and  navigation  through  the  virtual  environment.  It  also 
addresses  potential  anthropometric  (i.e.,  fit  and  feel)  problems  posed  by  wearing  an  HMD  for 
extended  periods  of  time.  The  analysis  also  looks  for  reported  instances  of  simulator  sickness  and 
attempts  to  correlate  those  reports  with  the  control-display  conditions  that  were  present  at  the 
time  of  symptom  onset. 

Second,  a  human  factors  analysis  should  utilize  a  sufficiently  broad  array  of  data  collection 
methodologies  so  that  all  of  the  relevant  aspects  of  human  functionality  are  covered.  That  is,  the 
analysis  should  examine  the  visual  requirements  of  the  user,  as  well  as  his/her  auditory,  cognitive, 
and  haptic  needs.  Moreover,  the  analysis  should  include  measures  that  distinguish  between 
operability  of  the  device  (i.e.,  is  it  as  easy  to  use  as  practically  possible)  versus  its  acceptance  by 
the  user  (Wickens,  1984).  We  explicitly  incorporated  all  of  these  features  into  our  human  factors 
analysis  of  the  AGSS,  except  for  acceptance.  Although  we  recognize  the  importance  of 
acceptance  for  the  complete  analysis  of  aircrew  training  devices,  we  only  tangentially  address  this 
issue  within  our  survey.  As  discussed  below,  it  is  a  research  issue  we  will  investigate  more 
explicitly  and  thoroughly  in  our  subsequent  AGSS  research  efforts. 

In  addition,  a  human  factors  analysis  should  survey  a  number  of  different  users  who  have 
had  experience  with  the  system.  It  should  include  a  mix  of  survey,  interview,  and  observational 
measures  to  ensure  that  data  on  input,  output,  controls,  displays,  and  processing  deficiencies  are 
identified.  The  analysis  should  include  measures  that  tap  into  both  the  device’s  ease  of  use  as  well 
as  its  effectiveness  in  achieving  the  users’  stated  training  objectives.  Our  analysis  of  the  AGSS 
incorporates  all  of  these  features. 

Scope  of  the  Human  Factors  Survey 

The  Human  Factors  Survey  Framework  depicted  in  Figure  2  is  designed  to  capture  the 
logic  underlying  our  presentation  of  human  factors  issues  associated  with  the  AGSS  and  its 
various  components.  In  addition,  the  framework  characterizes  our  thinking  regarding  ensuing 
research  efforts  related  to  the  acceptance  and  training  effectiveness  of  the  AGSS.  The  topics 
explicitly  covered  by  the  present  survey  are  included  within  the  shaded  box.  The  items  outside  of 
the  box  were  not  explicitly  explored  in  this  survey,  but  are  nonetheless,  relevant  to  the  overall 
assessment  of  the  AGSS  as  a  training  device.  A  description  of  the  survey  framework  follows. 

Moving  from  left  to  right,  we  start  with  the  assumption  that  the  device  was  built  according 
to  certain  design  specifications  which  met  government  standards  since  the  device  was  approved  as 
RFT  in  March  1996.  As  shown  in  the  figure,  these  design  specifications  are  applicable  to  each 
device  component,  Le.,  helmet,  trainee  stations,  IOS,  weapons  simulation,  and  visual  system. 
(The  specific  design  specifications  for  the  AGSS  are  not  described  in  this  report.  We  do, 
however,  provide  a  general  description  of  the  main  systems  of  the  AGSS  in  the  Method  Section.) 
Associated  with  each  device  component  will  be  a  host  of  human  factors  issues,  represented  in  the 
Human  Factors  boxes  to  the  right  of  each  device  component. 
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Training 

Effectiveness 


Figure  2.  Human.  Factors  Survey  Framework. 

Below  each  arrow  is  a  dotted  Method  of  Use  box  indicating  that  a  component’s  use  may 
affect  certain  human  factors  assessments.  For  example,  the  IOS  may  be  judged  as  highly  effective 
and  “user  friendly”  when  one  student  is  being  trained  in  an  AGSS  station,  but  this  assessment  may 
change  dramatically  if  there  are  two  or  three  students  being  trained  simultaneously  in  the  AGSS. 
Alternatively,  the  helmet  may  rarely  be  judged  as  problematic  when  the  device  is  used  for  short 
durations,  but  rated  much  lower  when  it  is  used  for  longer  time  periods.  The  Method  of  Use  box 
is  dotted  to  indicate  that  these  factors  may  or  may  not  affect  participants’  human  factors 
assessments.  When  describing  the  results  for  each  device  component,  we  will  first  discuss  the 
significant  human  factors  findings,  and  then  where  applicable,  the  effects  due  to  different  methods 
of  use. 


The  survey  also  covers  select  overall  device  assessments,  as  represented  by  the  two  small 
solid  boxes.  Physiological  Symptoms  and  Training  Capability.  We  assessed  the  occurrence  of  any 
unusual  physiological  symptoms  (e.g.,  dizziness,  blurred  vision,  etc.)  during  or  following  the  use 
of  the  AGSS.  In  addition,  as  a  lead-in  to  a  subsequent  planned  study  of  AGSS  training,  we  also 
examined  the  overall  training  capability  of  the  device  by  soliciting  SME  commentary  on  the 
capability  of  the  AGSS  to  train  selected  AG/S  skills. 

The  next  box  in  Figure  2  is  Acceptance,  which  includes  both  instructor  and  student 
acceptance  of  the  AGSS,  and  the  training  it  provides.  Acceptance  is  an  important  contributor  to 
the  training  effectiveness  of  this  (or  any)  device,  and  in  many  ways  reflects  the  sum  of  the 
previous  components  of  the  model.  That  is,  human  factors  judgments  of  the  various  components, 
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the  experience  of  unusual  physiological  symptoms,  and  its  training  capability  will  undoubtedly  be 
summed  and  weighted  in  subjective  ways  by  both  AGSS  students  and  instructors  to  influence  their 
acceptance  of  the  device.  We  also  realize,  as  depicted  in  the  box  below  Acceptance,  that  Other 
Factors  (e.g.,  preconceptions  about  simulators,  age,  rumors  about  device-associated  reductions  in 
flying  hours,  etc.)  will  affect  device  acceptance.  Acceptance  lies  outside  of  the  large  shaded 
dotted  box  because  it  is  not  explicitly  addressed  in  this  survey.  However,  some  participants  made 
direct  references  to  acceptance  in  their  comments,  and  we  cite  these  reports  where  appropriate  in 
the  results  section.  We  have  included  the  Acceptance  and  Training  Effectiveness  boxes  in  our 
model  as  these  are  the  next  two  research  areas  we  will  pursue  after  this  survey.  Training 
Effectiveness  is  shown  to  the  far  right  since  we  expect  that  it  will  reflect  all  of  the  preceding 
factors.  The  bi-directional  arrow  connecting  the  Acceptance  and  Training  Effectiveness  boxes 
acknowledges  the  likelihood  that  these  two  items  are  highly  interdependent. 

The  above  description  by  no  means  exhausts  all  of  the  factors  that  might  potentially  be 
included  in  an  analysis  of  the  utility  and  design  of  the  AGSS.  It  does,  however,  establish  the 
dominance  of  human  factors  issues  in  determining  the  training  effectiveness  of  the  AGSS  while 
also  providing  an  effective  context  for  understanding  the  survey  results. 

METHOD 

AGSS  System  Description 

The  AGSS  (see  Figure  3)  is  a  ground-based  trainer  designed  to  provide  both  basic  skills 
and  MR  training  to  USAF  AG/S  and  FE  personnel.  The  AGSS  is  a  reconfigurable  (MH-53J  and 
MH-60G)  device  with  a  three-degree-of-freedom  (pitch,  roll,  and  heave)  motion  base.  The  device 
simulates  sound,  recoil  vibration,  trajectory,  and  visual  imagery  for  both  the  .50  caliber  (cal)  and 
7.62  millimeter  (mm)  guns.  Gun  design  mimics  actual  gun  control,  position,  weight,  feel,  and 
operation.  All  ship,  gun,  and  air  sounds  are  heard  by  the  trainees,  including:  engines,  rotors, 
guns,  landing  gear,  warnings,  and  air  stream  effects.  The  visual  system  provides  both  day  and 
NVG  night  scenes  including:  outside  view,  inside  cabin,  gun  barrel,  fixed  and  moving  targets, 
weapon  effects,  effects  of  hits  or  misses,  and  environmental  conditions.  The  AGSS  is  designed  to 
operate  as  a  stand-alone  trainer  (AGSS  only),  an  integrated  trainer  (AGSS  with  a  host  cockpit, 
either  the  MH-53J  WST  or  the  MH-60G  WST),  or  networked  (AGSS  with  a  host  and  multiple 
training  devices). 


Figure  3.  Outside  View  of  the  AGSS. 


11 


The  main  systems  we  examined  in  the  survey  are  presented  in  Figure  4.  Although  this 
listing  does  not  include  all  of  the  systems  that  comprise  the  AGSS,  it  does  cover  those  of  greatest 
interest  to  the  end  user,  where  the  Trainee  Station  is  central.  In  addition,  systems  not  identified  in 
the  figure  may  be  captured  within  the  context  of  larger  applicable  systems  (e.g.,  aural  cues  (gun 
sounds)  within  Weapons  Simulation).  With  this  schematic  in  mind,  a  brief  description  of  each 
major  AGSS  system  follows. 


Figure  4.  Main  System  Schematic  of  the  AGSS. 


Helmet 

The  AGSS  is  equipped  with  three  helmets  for  the  left,  right,  and  tail  positions.  The  helmet 
is  a  Kaiser  Electro-Optics  Sim  Eye™  (see  Figure  5).  It  is  enclosed,  lightweight  (2  lbs),  and  fully 
adjustable  with  removable  velcro  pads  of  various  sizes.  Each  helmet  has  a  binocular  HMD  which 
is  attached  to  the  helmet  using  plastic  screws.  In  addition,  each  helmet  is  equipped  with  a 
magnetic  head-tracker  system.  The  electronic  and  communication  cables  attach  to  the  HMD  and 
extend  past  the  shoulders  and  lead  up  to  the  electronic  interlace  unit  (EIU)  in  the  center  of  the 
AGSS.  Each  helmet  also  has  headphones  and  a  microphone. 


Figure  5.  Kaiser  Electro-Optics  Sim  Eye™  AGSS  Helmet. 
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Trainee  Stations 


As  mentioned,  up  to  three  students  can  be  trained  simultaneously  in  the  AGSS.  Each 
trainee  station  provides  a  spatially  and  visually  correct  representation  of  the  aircraft  (either  the 
MH-53J  or  the  MH-60G,  depending  on  the  designated  configuration).  A  7.62  mm  or  a  .50  cal 
weapon  can  be  mounted  in  each  station.  The  visual  representation  of  the  station  is  transferred 
through  the  IG  and  contains  aircraft-specific  texturing  and  details.  All  window  frames  and  edges 
have  the  appropriate  spatial  relationship  to  the  gunner.  In  addition,  crew  seats  and  harnesses  are 
visually  and  functionally  realistic. 

Instructor  Operator  Station 

Instructors  are  located  outside  the  device  at  an  IOS  with  six  screens  (see  Figure  6):  one 
for  instructor  input  (lower  left);  one  for  information  output  (upper  left);  one  for  instructor 
viewpoint  selection,  either  phot’s  eye  view,  God’s  eye  view,  or  Target  Zoom  (lower  middle);  and 
three  that  repeat  each  trainee’s  visual  scene  (i.e.,  Left  Station,  upper  middle;  Right  Station,  upper 
right;  Tail  Station,  lower  right).  There  are  two  positions  at  the  IOS,  each  with  a  headset.  The 
primary  instructor  position  is  located  in  front  of  the  instructor  input  and  output  screens.  From 
this  position  the  instructor  is  capable  of  interacting  with  the  students  in  the  AGSS  through  the 
intercom.  The  second  position  is  in  front  of  the  student  repeater  screens.  The  instructor  or 
observer  at  this  station  can  watch  and  listen  to  all  of  the  training  activities,  but  he/she  cannot 
interact  directly  with  the  students  in  the  AGSS  through  the  intercom.  The  IOS  also  has  a  space 
ball — which  enables  the  instructor  to  position  targets,  fly  air-to-air  models,  and  fly  the  AGSS 
when  it  is  used  for  stand-alone  training — a  color  printer,  and  a  work  table.  Pre-programmed 
flight  paths  can  be  recorded  and  replayed  for  multiple  practice  sessions  on  the  same  flight  path. 
Performance  monitoring  pages  allow  trainers  to  print  out  gunnery  performance  scores  (e.g., 
number  of  hits,  type  of  ammunition,  etc.)  for  individual  gunners  as  well  as  gunner  crews. 


Figure  6.  AGSS  Instructor  Operator  Station. 
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Weapons  Simulation 

The  AGSS  is  capable  of  simulating  sound,  recoil,  vibration,  trajectory,  and  visual  imagery 
for  both  the  .50  cal  and  the  7.62  mm  guns  (see  Figure  7).  The  AGSS  duplicates  all  gun  controls, 
positions,  weight,  feel,  and  operations  of  each  gun.  High  fidelity  airstream  effects  vary  with 
airspeed  and  gun  orientation.  These  airstream  effects  are  provided  through  individual  control 
loading  mechanisms  which  support  all  three  gun  positions. 


Figure  7.  Left  Aerial  Gunner  Station  with  7.62  mm  Weapon  Mounted  in  the  Window. 
Visual  System 

The  primary  components  of  the  visual  system  are:  the  HMD,  the  head-tracker  system,  and 
the  IG.  The  HMDs  are  full-color  and  support  a  60°  by  40°  FOV  with  1028  by  1024  resolution. 
The  head-tracker  system  has  an  eight-foot  radius  and  provides  head  position  and  attitude 
information  to  the  IG  and  host  computer.  There  is  one  transmitter  with  a  receiver  on  each  of  the 
three  helmets.  The  SE  2000  IG  is  used  when  the  AGSS  is  in  stand-alone  mode.  When  the  AGSS 
is  in  network  mode,  other  IGs  available  at  the  58  TRSS,  like  Compuscene  V,  may  be  used.  The  IG 
provides  photo-specific  scenes  to  the  HMDs.  The  visual  simulation  includes  interior  aircraft 
features  (e.g.,  cabin,  weapon  barrels,  etc.)  and  exterior  aircraft  features  (e.g.,  rotors).  Further,  the 
out-of-the-window  depiction  includes:  air,  sea,  terrain  scenery,  targets,  and  weapons  effects,  i.e., 
tracer  path  and  bullet  impact  (Reed,  1996). 


Subjects 

Eleven  experienced  rotary-wing  instructors  volunteered  to  participate  in  this  survey;  five 
FEs  and  six  AGs.  Ten  were  crewmembers  from  the  MH-53J  weapon  system  and  one  was  from 
the  MH-60G.  Seven  were  active  duty  USAF  and  four  were  Lockheed  Martin  instructors.  All 
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survey  participants  had  extensive  instructor  experience  in  their  respective  positions.  In  addition, 
every  participant  had  some  level  of  experience  with  the  AGSS.  A  summary  of  the  participants’ 
AGSS  and  flying  experience  appears  in  Table  1. 


Table  1.  Summary  of  Survey  Participants’  AGSS  and  Flying  Experience. 


- 

Position 

Flight  Engineer  (n=5) 

Aerial  Gunner  (n=6) 

Mean 

Maximum 

Minimum 

Mean 

Maximum 

Minimum 

AGE 

40 

48 

32 

36 

43 

28 

AGSSTOT 

34.80 

100.00 

2.00 

114.33 

300.00 

1.00 

AGSSBOX 

5.70 

12.00 

.50 

19.33 

50.00 

1.00 

COMBAT 

226.80 

433.00 

.00 

86.17 

280.00 

.00 

FLYTIME 

4520.00 

6500.00 

1800.00 

1720.00 

2500.00 

720.00 

FLYTLM 

8.00 

20.00 

.00 

19.17 

40.00 

.00 

FLYTNVG 

1420.00 

2500.00 

900.00 

1141.67 

2000.00 

500.00 

Note.  AGSSTOT  =  estimated  total  number  of  hours  experience  with  the  AGSS,  either  during  development  or 
training;  AGSSBOX  =  estimated  number  of  hours  experience  at  an  AGSS  gunner  station;  COMBAT  =  estimated 
number  of  combat  hours;  FLYTIME  =  estimated  total  number  of  flying  hours;  FLYTLM  =  estimated  number  of 
flying  hours  in  the  month  prior  to  the  survey,  and  FLYTNVG  =  estimated  number  of  flying  hours  using  NVGs. 

The  Survey 

The  survey  was  designed  to  capture  both  quantitative  and  qualitative  information  about 
the  functionality  and  fidelity  of  the  AGSS.  A  brief  description  of  the  data  collection  tool  follows, 
and  an  abbreviated  version  of  the  survey  appears  in  Appendix  A.  The  survey  was  divided  into 
nine  sections:  (1)  background  information,  (2)  helmet,  (3)  trainee  stations,  (4)  IOS,  (5)  weapons 
simulation,  (6)  instructor/student  interaction,  (7)  visual  system,  (8)  training  capability,  and  (9) 
miscellaneous. 

The  background  information  section  included  questions  regarding  the  participant’s  age, 
current  responsibilities,  crew  position,  AGSS  experience,  flying  experience,  visual  history,  and 
past  experience  with  unusual  physiological  symptoms  (e.g.,  dizziness  or  blurred  vision)  during 
flight,  after  flight,  and  during  everyday  activities  (e.g.,  reading). 

Representing  a  primary  component  of  the  AGSS,  the  helmet  section  of  the  survey  was 
fairly  detailed  and  thorough  including  subsections  on  helmet  component  breakage  and 
malfunctions,  comfort  and  fit  problems,  the  occurrence  of  problems  with  the  electronic  and 
communication  cables,  and  adjustment  requirements.  For  both  the  helmet  breakage  and  comfort 
and  fit  subsections,  specific  helmet  components  (e.g.,  chinstrap  or  visor)  were  itemized  for 
ratings.  A  5-point  scale  (never,  rarely,  sometimes,  often,  always)  was  provided.  Space  was  also 
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provided  beneath  each  question  so  that  the  interviewer  could  write  the  details  of  subject 
responses,  including  under  what  circumstances  any  breakage  occurred,  suggestions  about 
remedying  noted  problems,  and  ideas  that  participants  had  about  improving  the  system.  In  the 
cable  subsection,  specific  yes/no  questions  were  listed  about  whether  the  cables  interfered  with  or 
restricted  task  performance.  The  adjustments  subsection  provided  space  for  specific  time 
estimates  required  for  fitting  the  helmet  properly  and  aligning  the  visuals  (during  the  subject’s  first 
and  subsequent  exposures),  and  whether  any  readjustments  were  required  during  AGSS  sessions. 
At  the  end  of  the  helmet  section,  space  was  provided  for  general  comments. 

The  trainee  station  section  included  blanks  for  rating  trainee  station  components,  e.g., 
gun  set-up,  station  mapping,  etc.  A  5-point  Likert  Scale  1  (poor)  to  5  (exceptional)  was 
provided.  The  verbal  anchors  of  the  scale  referred  to  both  similarity  of  the  station  component  to 
the  aircraft  (the  MH-53J  or  MH-60G  as  applicable)  and  the  training  capability  of  the  component 
(see  Table  2).  Space  was  again  provided  for  comments  and/or  suggestions. 


Table  2.  Survey  Scale  for  Rating  Judged  Similarity  to  Aircraft  and  Training  Capability. 


Rating 

Value 

Definition 

1 

Poor 

No  similarity  between  simulator  and  aircraft.  Provides  negative 
training  and  has  major  deficiencies. 

2 

Min.  Required 

Little  similarity  between  simulator  and  aircraft.  Major 
modification  is  required  to  provide  adequate  training  capability 

3 

Standard 

Acceptable  similarity  between  simulator  and  aircraft.  Adequate 
training  is  provided. 

4 

Outstanding 

The  simulator  and  the  aircraft  similarity  is  nearly  equal. 

Training  capability  is  similar  to  that  provided  in  the  aircraft. 

‘  5 

Exceptional 

The  simulator  and  the  aircraft  are  virtually  the  same.  Training 
capability  is  beyond  expectations. 

Like  the  helmet  section,  the  IOS  section  was  fairly  detailed  and  inclusive.  The  first 
subsection  contained  questions  about  the  location  of  the  IOS  outside  the  AGSS  and  the  level  of 
task  saturation  experienced  by  operators.  The  5-point  Never  to  Always  scale  was  used,  and  space 
was  provided  for  comments.  The  main  IOS  subsection  included  specific  components  of  the  IOS 
(e.g.,  the  space  ball)  which  were  to  be  rated  on  their  training  effectiveness  using  a  1  (poor)  to  5 
(exceptional)  point  Likert  Scale  (see  Table  3). 


Table  3.  Survey  Scale  for  Rating  Judged  Training  Effectiveness. 


Rating 

Value 

Definition 

1 

Poor 

Cannot  train  required  tasks;  system  has  major  deficiencies. 

2 

Min.  Required 

Minimal  training  can  be  provided;  major  modification  is 
required  to  provide  adequate  training  capability. 

3 

Standard 

Training  capability  is  acceptable. 

4 

Outstanding 

Most  of  the  required  tasks  can  be  trained;  few,  if  any, 
modifications  required. 

5 

Exceptional 

Training  capability  is  above  expectations,  and  tasks  beyond 
those  required  can  be  trained 
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The  weapons  simulation  section  duplicated  the  format  of  the  trainee  station  section,  i.e., 
specific  components  were  rated  from  1  (poor)  to  5  (exceptional),  using  the  scale  illustrated  in 
Table  2.  Each  component  was  rated  separately  by  gun  type  (i.e.,  .50  cal  and  7.62  mm)  to  account 
for  any  judged  differences  between  the  two. 

The  instructor/student  interaction  section  was  very  brief.  It  included  six  yes/no 
questions  about  instructor  ability  to  communicate  with  students  from  the  IOS,  and  student  ability 
to  communicate  with  instructors  and  each  other  while  in  the  AGSS. 

The  visual  system  section  had  two  main  subsections.  The  first  was  a  repeat  of  the 
physiological  symptoms  portion  of  the  background  information  section  with  the  AGSS  as  the 
context,  e.g..  Did  any  of  the  following  symptoms  (e.g.,  dizziness)  occur  during  or  after  training  in 
the  AGSS?  This  subsection  was  followed  by  a  number  of  visual  system  elements  and  qualities 
(e.g.,  visual  contrast)  with  space  provided  for  judged  ratings  and  comments  on  both  day  and  NVG 
night  training  quality.  Due  to  limited  variability  between  the  day  ratings  and  the  NVG  night 
ratings,  we  primarily  used  the  day  ratings.  However,  some  exceptions  (e.g.,  responses  related  to 
day  and  night  FOV)  are  noted  in  the  text. 

The  training  capability  section  included  39  required  AG  and  FE  skills  that  were  selected 
based  on  analysis  and  discussions  with  AG  and  AGSS  SMEs.  Space  was  provided  for 
participants’  1  (poor)  to  5  (exceptional)  assessments  (see  Table  4)  and  comments  on  the  training 
capability  of  the  AGSS  for  the  skills  under  both  day  and  NVG  night  conditions. 

Table  4.  Survey  Scale  for  Rating  Judged  Training  Capability  for  Specific  AG/S  Skills. 


Rating 

Value 

Definition 

1 

Poor 

Cannot  train  this  skill  in  AGSS. 

2 

Min.  Required 

Only  minimal  training  on  this  task  provided  in  the  AGSS. 

3 

Standard 

Acceptable  training  provided  on  this  task  in  AGSS. 

4 

Outstanding 

Training  provided  on  this  task  is  above  expectations. 

5 

Exceptional 

Training  on  this  task  is  beyond  expectations  due  to  capabilities  of 
the  AGSS  not  available  in  the  aircraft. 

The  final  section  was  a  miscellaneous  section.  It  included  questions  on  the  overall  feel  of 
the  AGSS,  the  motion  system,  and  whether  subjects  experienced  any  disorientation  in  the  AGSS. 

Data  Collection  Procedures 

The  survey  was  conducted  using  one-on-one  guided  interviews.  All  of  the  interviews 
were  conducted  by  the  primary  author.  The  process  began  with  both  written  and  oral  instructions 
to  the  subjects,  including  information  about  the  confidentiality  and  purpose  of  the  survey.  Each 
subject  read  and  signed  a  statement  of  informed  consent.  The  participants  were  then  asked  to 
make  their  ratings  and  comments  based  on  their  subject-matter  expertise  and  their  experiences 


17 


with  the  AGSS.  The  interviewer  stressed  the  importance  of  making  candid  device  assessments 
and  the  fundamental  goal  of  improving  the  system  and  its  use  by  students  and  instructors. 

Each  interview  was  conducted  using  the  same  format  and  following  the  same  topical  order, 
i.e.,  from  the  background  information  section  through  the  miscellaneous  section.  However,  some 
sections  and/or  questions  were  skipped  if  the  participant  did  not  have  experience  with  the  device  in 
that  capacity  (e.g.,  as  an  instructor/operator  or  using  the  device  under  NVG  conditions).  The 
interviewer  prompted  comments  associated  with  extreme  rating  values.  Responses  were  written 
directly  on  the  survey  by  the  interviewer.  Most  interviews  lasted  1-2  hrs  (the  minimum  was  40 
min  and  the  maximum  1 17  min),  depending  on  the  subject’s  amount  of  experience  with  the  device. 
Those  with  more  device  experience  tended  to  take  longer  to  interview. 

Choice  of  Analysis 

The  survey  contained  three  types  of  quantitative  responses:  (a)  yes/no  responses; 
(b)  never,  rarely,  sometimes,  often,  always  scale  ratings;  and  (c)  1  (poor)  to  5  (exceptional)  Likert 
Scale  ratings.  In  addition,  each  participant  provided  researchers  with  a  substantial  amount  of 
qualitative  information,  including  their  informed  opinions  about  device  components  or  features, 
particular  problems,  and  often  suggested  solutions.  There  were  very  few  questions  that  simply 
required  a  yes/no  response,  and  for  these  we  simply  report  frequency  counts. 

For  the  scaled  responses,  we  were  faced  with  more  complex  analytical  issues.  There  were 
two  critical  challenges  that  had  to  be  met.  One,  the  quantitative  information  had  to  be  logically 
linked  to  the  qualitative  information.  Two,  the  selected  analysis  or  analyses  had  to  reflect  both  the 
positive  and  negative  comments  and  ratings  concerning  the  AGSS.  This  was  necessary  to  ensure 
that  the  evaluation  would  yield  a  fair,  impartial  assessment  of  the  device’s  capacity  and  worth  to 
the  training  community. 

In  order  to  accomplish  these  goals,  we  selected  two  statistical  techniques  to  analyze  our 
rating  data,  one  for  the  AGSS  component  ratings  and  the  other  for  the  AGSS  training  capability 
ratings.  First,  we  chose  the  Kolmogorov-Smimov,  one-sample,  goodness-of-fit  test  (Hays,  1988; 
Siegel,  1956)  to  analyze  AGSS  system  component  ratings.  The  Kolmogorov-Smimov  is  a 
nonparametric  test  of  the  extent  to  which  an  observed  distribution  of  scores  deviates  from  some 
expected  distribution.  F or  each  of  our  analyses,  we  used  the  Kolmogorov-Smimov  analysis  to  test 
whether  the  obtained  rating  distribution  was  significantly  different  from  a  uniform  rectangular 
distribution.  This  analysis  lets  us  determine  whether  the  observed  ratings  for  a  given  survey  item 
were  significantly  bunched  toward  the  positive  end  of  the  scale.  Importantly,  the  Kolmogorov- 
Smimov  technique  preserves  the  ordinal  properties  of  the  original  rating  scale,  making  it  a  more 
powerful  test  than  the  traditional  chi-square  test.  In  addition,  due  to  the  nature  of  the  scale  use  by 
our  participants,  i.e.,  their  limited  use  of  extreme  scale  values,  we  chose  to  collapse  the  scale 
responses  into  three  categories.  We  combined  the  “Is”  and  “2s”  into  a  single  category  and  we 
combined  the  4s  and  “5s”  into  a  single  category.  The  Kolmogorov-Smimov  analysis  could  still 
be  applied  to  these  data  without  any  loss  of  statistical  power.  Comparisons  were  done  against  a 
fairly  liberal  alpha-level  of  .20. 
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Second,  hierarchical  cluster  analysis  was  used  to  analyze  the  training  capability  ratings. 
The  primary  goal  of  cluster  analysis  is  to  identify  homogenous  groups  of  objects,  in  this  case 
skills,  based  on  distances  among  the  objects.  We  used  hierarchical  clustering  analysis  to  identify 
distinct  skill  clusters.  The  main  steps  of  this  approach  were:  (a)  transforming  the  rating  values 
provided  by  the  subjects  into  z-scores,  thereby  appropriately  weighting  each  participant’s  use  of 
the  scale;  (b)  calculating  the  distances  between  all  possible  pairs  of  skills;  and  (c)  combining  the 
skills  into  clusters  or  groups.  In  hierarchical  cluster  analysis,  all  objects  begin  as  separate  clusters 
where  there  are  as  many  clusters  as  there  are  objects.  At  each  iteration  (or  step)  of  the  analysis, 
an  object  is  added  to  an  individual  cluster  or  two  clusters  are  combined  (Norusis,  1993).  There 
are  several  methods  for  combining  clusters.  We  chose  the  between-group  linkage  approach  which 
ensures  that  at  each  step  cases  are  combined  so  that  the  average  distance  between  all  cases  in  the 
resulting  cluster  is  as  small  as  possible. 

RESULTS 
Overall  Findings 

Before  discussing  the  specific  results  from  each  section  of  the  survey,  we  present  an 
overview  of  our  major  findings  (see  Appendix  B  for  Raw  Rating  Data).  Table  5  summarizes  the 
results  from  the  five  major  AGSS  systems  assessed  and  the  overall  assessment  of  the  device’s 
Training  Capability.  Besides  the  average  overall  rating  which  are  generally  acceptable,  the  table 
gives  our  general  assessment  (inferred  by  the  researchers  from  participants’  comments  and 
ratings)  and  representative  comments  for  each  system. 

Table  5.  Summary  Assessments  for  Each  Major  Survey  Section. 


System 

■SEII 

Assessment 

Representative  Comments 

Helmet 

4.3 

Good 

(when  working) 

=>  When  properly  fit,  everything  should 
be  okay. 

Trainee  Stations 

3.8 

Good 

=>  Much  better  than  first  design. 

Instructor  Operator 
Station 

3.9* 

Good 

(room  for  improvement) 

=>  Overall,  it  is  nice. 

=>  Ownship  flying  with  space  ball  is 
worthless. 

Weapons  Simulation 

3.6 

Needs  Work 

=>  Not  enough  tracers  per  rounds  fired. 

=>  Lead  and  lag  problem. 

Visual  System 

3.2 

Faulty 

=>  Can’t  see  targets  until  they  are  too 
close  even  with  instructor  input 

Training  Capability 

3.5 

Good 

(Qualified) 

=>  Primary  impacts  will  be  on  crew 

coordination,  scanner  calls,  and  target 
acquisition  if  visual  problems  fixed. 

*Note.  Represents  an  inversion  of  the  subject  responses  (i.e.,  never  to  always)  to  make  it  parallel  to  other 
component  ratings. 

Starting  with  the  helmet,  we  see  that  its  overall  rating  was  quite  high,  making  it  the  highest 
rated  AGSS  system.  From  these  ratings,  we  gave  it  a  general  assessment  of  “good.”  However, 
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low  ratings  for  certain  essential  parts  of  the  helmet,  as  well  as  participant  criticisms,  required  us  to 
include  the  “when  working”  caveat  in  our  assessment.  As  discussed  more  completely  below, 
critical  components  of  the  helmet  (e.g.,  the  head-tracker  and  CRTs)  are  subject  to  frequent 
breakage  and/or  malfunctions.  Moreover,  due  to  the  newness  of  the  device,  the  fitting  and 
aligning  processes  are  fallible  which  can  lead  to  additional  helmet  problems  (e.g.,  discomfort). 

The  mean  rating  for  the  helmet  was  determined  in  a  different  manner  than  the  other 
systems.  For  the  trainee  stations,  IOS,  weapons  simulation,  visual  system,  and  training  capability, 
we  asked  participants  to  rate  each  system  overall  and  averaged  their  responses  to  this  question. 
In  order  to  provide  the  overall  mean  rating  that  appears  in  Table  5,  we  summed  participant 
responses  to  each  of  the  helmet  questions  and  divided  by  the  number  of  questions.  We  believe 
this  measure  provides  an  appropriate  overall  SME  assessment  of  the  helmet. 

The  trainee  stations  received  a  fairly  high  overall  average  rating.  Indeed,  participants’ 
comments  were  consistent  with  this  high  rating.  It  is  perhaps  a  subtle  distinction,  but  the  majority 
of  the  comments  focused  on  issues  that  could  make  the  trainee  stations  better ,  rather  than  flaws 
about  the  design  and  functionality  of  the  stations  themselves,  leading  us  to  the  unqualified 
assessment  of  “good.” 

The  IOS  had  a  slightly  higher  overall  mean  rating  than  the  trainee  stations.  However,  we 
assessed  it  as  “good”  with  a  “room  for  improvement”  qualifier  because  particular  features 
(discussed  later)  were  given  unacceptable  ratings  by  participants.  In  addition,  although  some  of 
the  comments  were  simple  suggestions  for  improving  the  training  capability  of  the  IOS  (as  we 
saw  with  the  trainee  station  comments),  many  focused  on  changes  that  would  enable  training. 

The  overall  “needs  work”  assessment  of  weapons  simulation  came  primarily  from  the 
frequently  cited  modeling  deficiencies  of  the  bullet  path  (described  later).  The  fairly  high  overall 
average  rating  of  weapons  simulation,  despite  this  rather  major  problem,  stems,  we  think,  from 
the  feet  that  many  of  the  gunnery-related  training  capabilities  of  the  AGSS  simply  did  not  exist 
prior  to  its  introduction 

Survey  participant  comments  and  the  somewhat  lower  ratings  led  to  an  overall  assessment 
of  the  visual  system  as  “faulty.”  Participants  noted  the  tremendous  importance  of  visual  input  for 
many  AG/S  tasks,  and  given  the  current  quality  of  the  AGSS  visual  system,  the  level  of  training 
that  could  be  provided  by  the  AGSS  for  many  AG/S  tasks  was  severely  limited. 

Finally,  overall  training  capability  of  the  AGSS  was  assessed  as  “good”  with  qualifications. 
That  is,  participant  comments  and  ratings  reflected  overwhelming  agreement  that  the  training 
potential  of  the  AGSS  was  tremendous,  especially  in  the  areas  of  crew  coordination,  target  ident¬ 
ification,  and  target  acquisition.  However,  at  least  in  terms  of  the  latter  two  skills,  the  potential 
could  not  be  fully  realized  until  (as  mentioned)  improvements  were  made  to  the  visual  system. 

To  further  substantiate  the  average  ratings  and  the  general  assessments  of  each  AGSS 
system,  provided  in  Table  5,  we  have  divided  the  rest  of  the  survey  results  into  two  main  sections: 
a  component  rating  analysis  and  a  qualitative  summary  of  participant  comments.  The  component 
rating  analysis  unveils  both  the  positively  and  negatively  rated  features  of  the  AGSS.  In  addition. 
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this  analysis  reveals  features  of  the  AGSS  that  received  “standard”  or  “acceptable  ratings  (i.e., 
neither  significantly  positive  nor  negative)  that  in  conjunction  with  the  participant  comments 
(detailed  in  the  next  section)  suggest  several  AGSS  features  which  may  require  modifications. 
Briefly,  the  ratings  and  qualitative  analysis  illustrate  that:  (a)  most  features  of  the  AGSS  are 
acceptable  for  training,  but  may  benefit  from  improvement;  (b)  quite  a  few  AGSS  features  are 
superb;  and  (c)  the  AGSS,  overall,  functions  fairly  well  as  an  AG/S  training  medium. 

Analysis  of  System  Ratings 

As  explained  in  the  Choice  of  Analysis  section,  we  chose  the  Kolmogorov-Smimov  one- 
sample  test  for  goodness  of  fit  (Hays,  1988;  Siegel,  1956)  to  analyze  the  component  rating  data 
for  each  system.  Because  this  technique  retains  the  ordinal  nature  of  the  data  inherent  in  the 
rating  scale,  it  is  more  powerful  than  the  traditional  chi-square  statistic.  It  also  allows  us  to 
combine  data  from  adjacent  cells  (i.e.,  the  “1”  and  “2”,  and  “4”'  and  “5”  ratings)  without 
sarrifiring  statistical  power.  We  have  organized  the  component  ratings  results  into  five 
subsections  corresponding  to  each  of  the  AGSS  systems — Helmet,  Trainee  Stations,  IOS, 
Weapons  Simulation,  and  Visual  System. 

Each  subsection  is  organized  around  a  table  that  depicts  the  results  of  the  Kolmogorov- 
Smimov  one-sample  test.  Within  each  table,  we  indicate  the  system  component  that  is  rated,  the 
number  of  participants  who  rated  the  component  (N),  the  maximum  deviation  (Max.  D)  between 
the  expected  and  observed  cumulative  probability  distributions,  and  the  two-tailed  probability 
level  for  that  deviation.  When  a  probability  level  is  not  stated,  this  means  that  the  ratings 
associated  with  that  component  were  neither  significantly  positive  nor  significantly  negative.  In 
other  words,  participants  provided  “acceptable”  ratings  for  these  components.  These 
components,  as  well  as  the  features  receiving  significantly  negative  ratings,  are  discussed  further 
in  our  qualitative  analysis  of  participant  comments. 

It  should  be  noted  that  the  hems  associated  with  negative  deviations  (though  still 
statistically  acceptable)  are  also  identified  through  this  analysis.  Given  the  level  of  development  of 
the  device  (i.e.,  judged  RFT  for  stand-alone  training),  it  stands  to  reason  that  many  components 
are  exceptional,  while  a  majority  of  them  are  acceptable ,  but  could  be  improved.  Perhaps  the 
most  remarkable  finding,  however,  given  the  one-of-a-kind  nature  of  the  AGSS  and  the  novel  use 
of  VR  technology  in  the  AG/S  training  realm,  is  the  fact  that  only  two  components  of  the  AGSS 
were  rated  significantly  poor.  We  briefly  describe  the  results  of  the  rating  analyses  for  each 
system  below. 

Helmet 

Collapsing  across  two  subsections  of  the  survey  (helmet  malfunctions  and  helmet  comfort 
and  fit),  17  helmet  characteristics  were  rated  for  frequency  of  problem  occurrence.  All  but  four 
characteristics  had  low  levels  of  reported  problem  occurrence,  i.e.,  were  significantly  positive  (see 
Table  6). 
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Table  6.  Helmet  Rating  Analysis. 


Component 

N 

Max.  D 

Prob. 

11 

.670 

.01 

11 

.488 

.01 

11 

.579 

.01 

Comm.  Cable 

11 

.379 

.05 

CRTs 

11 

.215 

Electronics  Cable 

11 

.397 

.05 

i  mmtmssm 

11 

.397 

.05 

11 

.670 

.01 

General  Comfort 

11 

.397 

.05 

Head-tracker 

11 

.215 

Helmet  Pads 

11 

.488 

:01 

Hot  Spots 

11 

.306 

11 

.579 

.01 

Helmet  Fit 

11 

306 

11 

.397 

.05 

Unbalanced 

11 

.579 

.01 

Visor 

IT 

.488 

.01 

The  four  bolded,  italicized  components — CRTs,  head-tracker,  hot  spots,  and  helmet  fit — 
failed  to  receive  reliably  positive  ratings  from  the  Kolmogorov-Smimov  test.  As  a  result,  we  will 
focus  on  these  areas  when  discussing  participant  critiques  and  recommendations  for  the  helmet  in 
the  next  section. 

Trainee  Stations 

Of  the  six  trainee  station  components  shown  in  Table  7,  only  one  (gun  feel)  received  a 
reliably  positive  rating.  The  others  failed  to  achieve  significance  in  either  direction,  and,  as  such, 
achieved  “standard”  ratings  (with  positive  trends).  As  the  comments  will  illustrate,  the  current 
development  of  the  trainee  stations  is  acceptable  for  training,  but  participants  make  many 
insightful  remarks  regarding  potential  improvements. 


Table  7.  Trainee  Stations  Rating  Analysis. 


Component 

N 

Max.  D 

Prob. 

Gun  Feel 

11 

.667 

.01 

Gun  SetUp 

11 

.242 

Mobility  in  Station 

11 

.242 

VR.A/C  Interior 

11 

.303 

VR  Other  Stations 

11 

.242 

Station:  Overall 

11 

.302 

Note.  Crew  seats  and  crew  harnesses  were  not  included  in  this  analysis  due  to  the  few  number  of  participants  who 
had  used  these  components  in  the  AGSS.  They  are,  however,  discussed  in  our  qualitative  summary 
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IOS 


As  shown  in  Table  8,  the  IOS  runs  the  gambit  of  possible  ratings.  Several  features  were 
rated  significantly  positive  (instructor  screen,  layout;  crew  performance  page;  gunner  performance 
page;  and  IOS  overall),  one  was  rated  significantly  negative  (the  space  ball  for  ownship  flying), 
and  several  features  were  rated  “acceptable.”  In  addition,  it  should  be  noted  that  while  all  the 
other  components  that  were  judged  “acceptable”  to  this  point  have  had  positive  mavimnm 
deviations,  the  second  instructor  position  of  the  IOS  is  the  first  acceptably  judged  component  to 
have  a  negative  deviation.  This  may  indicate  that  the  second  instructor  position  needs  more 
improvement  than  the  other  acceptably  rated  components.  Later,  we  shall  see  that  comments 
from  participant  substantiate  this  interpretation. 


Table  8.  IOS  Rating  Analysis. 


Component 

N 

Max.  D 

Prob. 

Instructor  Screen,  Content 

8 

.295 

Instructor  Screen,  Layout 

8 

.420 

.10 

Second  Instructor  Position 

7 

-.238 

Space  Ball  Models 

8 

.295 

9 

M® 

Station  Layout 

8 

.333 

Student  Screens,  Content 

8 

.295 

Student  Screens,  Layout 

8 

.303 

PAGES: 

Crew  Performance 

8 

.545 

.10 

Flare  Monitoring 

7 

.333 

Ground  Track  Map 

8 

.333 

Gunner  Performance 

8 

.420 

.01 

Threat  Status 

7 

333 

IOS:  Overall 

8 

.420 

.10 

Weapons  Simulation 

Weapons  simulation  was  assessed  for  both  the  .50  cal  and  the  7.62  mm  guns. 
Participants  ratings  did  not  vary  across  the  gun  types,  so  each  feature  is  simply  treated  as  an 
assessment  of  the  general  weapons  simulation  of  the  AGSS.  Twelve  individual  aspects  of  AGSS 
weapons  simulation  as  well  as  gun  simulation  overall  were  rated  in  terms  of  their  fidelity  and 
training  effectiveness. 

As  shown  in  Table  9,  analyses  of  the  weapons  simulation  revealed  a  wide  range  of  feature 
rating  distributions:  five  subsystems  were  rated  significantly  positive  (including  guns  overall), 
three  were  rated  acceptable  with  positive  trends,  and  four  were  rated  acceptable  with  negative 
trends  (indicated  in  the  table  by  having  gray-shaded  Max.  D  cells).  As  discussed  in  the  next 
section,  the  latter  four  gun  features  (gun  alignment,  gun  pivot  point,  target  acquisition,  and 
tracers)  were  the  primary  focus  of  participant  criticism  in  terms  of  weapons  simulation. 
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Table  9.  Weapons  Simulation  Rating  Analysis. 


Component 

N 

Max.  D 

Prob. 

53  Gun  Position 

.266 

.15 

Air  Stream  Effects 

li 

.302 

Ballistics 

n 

.243 

Kciff  1WJ 1  HjftC '  JUMB 

n 

Gun  Controls 

li 

.575 

.01 

Gun  Pivot  Point 

n 

Ilf'*"'  •  - 

Gun  Recoil 

10 

.333 

.20 

Gan  Sounds 

li 

.333 

.15 

n 

.302 

fcg— 

ii 

Tracers 

li 

* 

Guns:  Overall 

n 

.333 

.15 

Note.  Gun  Position  60  was  not  included  in  this  analysis  due  to  the  few  number  of  participants  qualified  to  rate  this 
feature. 

Visual  System 

Twenty-four  specific  elements  of  the  visual  system  were  rated  for  their  training  utility 
under  both  day  and  NVG  night  conditions.  The  differences  in  participants’  ratings  of  the  visual 
system  did  not  vary  between  day  and  NVG  night  use,  and  therefore,  are  reported  as  general 
assessments  of  each  feature. 

The  visual  system  was  clearly  the  AGSS  system  that  was  associated  with  the  greatest 
amount  of  participant  dissatisfaction.  Three  points  from  Table  10  endorse  this  conclusion.  One, 
ratings  for  object  detection,  a  key  requirement  for  numerous  AG/S  tasks  (e.g.,  friend  vs  foe 
distinctions),  were  reliably  negative.  Two,  six  other  visual  system  features,  although, 
“acceptable,”  show  negative  deviations.  (These  are  indicated  in  the  table  by  having  gray-shaded 
Max.  D  cells.)  Three,  only  5  of  24  features  achieved  significantly  positive  ratings.  Of  these,  four 
were  at  the  most  liberal  probability  level  used  in  our  survey  analysis.  The  visual  system  was  also  a 
source  of  many  participant  comments  and  critiques  which  are  detailed  in  the  next  section. 

Table  10.  Visual  System  Rating  Analysis. 


Component 

N 

Max.  D 

Prob. 

Alignment 

11 

.148 

Brightness 

11 

.330 

20 

Contrast 

11 

.057 

Distance  Perception 

11 

.125 

FOV 

11 

.330 

.20 

Image  Clarity 

11 

.215 

11 

.148 

11 

.306 

.20 

WBSSBBi 

HHSH^  I  -  a- 
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Table  10.  Concluded. 


Object  Resolution 

11 

.239 

Scene  Clarity 

11 

.057 

Scene  Completeness 

11 

.239 

Scene  Continuity 

11 

.148 

Scene  Depth. 

11 

.215 

SPECIFIC  OBJECTS: 

747 

9 

330 

.20 

Explosions 

11 

.488 

.01 

Hind-D 

HUH 

.170 

Jeep 

'  ’ .  eshb 

Roads 

li 

HU  •  HH 

Runways 

10 

.270 

SAMs 

9 

■IH '  IIIIHI 

Trees 

11 

Truck 

9 

iBHenaHH 

Visual  Resolution 

11 

n.oa:-::*. ■ 

Visual  System:  Overall 

11 

.239 

Problem  Areas:  Comments  and  Recommendations  for  Improvement 

This  section  builds  on  the  previous  section  by  discussing  the  participant  comments 
associated  with  each  of  the  features  identified  in  the  Kolmogorov-Smimov  analysis  as  having 
either  negative  or  “acceptable”  ratings.  We  list  the  number  of  participants  reporting  problems 
with  a  feature  “sometimes”  or  more  when  the  participants  were  required  to  use  this  scale.  In 
terms  of  our  5-point  Likert  scale,  we  report  the  number  of  participants  (typically  out  of  1 1)  who 
rated  a  given  component/feature  as  a  “3”  or  less.  We  then  report  some  of  the  criticisms  cited  for 
each  component  discussed,  followed  by  recommendations  for  improvement. 

The  best  match  between  rating  scores  and  the  frequency  and  content  of  comments  was 
achieved  by  grouping  “3”  ratings  with  the  more  negative  “1”  and  “2”  ratings  (as  described)  and 
using  a  liberal  Kolmogorov-Smimov  criterion  of  p  <  .20.  This  was,  therefore,  adopted  as  our 
decision  rule  for  including  topics  in  this  section.  The  tendency  to  rate  an  area  as  a  “3”  (or 
acceptable)  and  still  provide  negative  comments  may  reflect  the  RFT  status  of  the  AGSS  which 
connotes  that  “standard”  (or  “3”)  criterion  levels  have  already  been  officially  met.  Additionally, 
there  were  several  survey  questions  associated  with  the  helmet  and  the  IOS  which  were  not 
amenable  to  the  Kolmogorov-Smimov  analysis;  the  responses  and  statistics  related  to  these 
questions  are  also  reported  below. 

Helmet 

Seventeen  aspects  of  the  helmet  were  explored  for  either  malfunctions  or  problems  with 
comfort  and  fit.  We  also  asked  several  questions  about  problems  associated  with  the  helmet 
communication  and  electronic  cables  and  the  required  helmet  donning  time. 
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Helmet  Components 


Comments.  Seven  helmet  components  were  rated  according  to  frequency  (from  “never 
occurred”  to  “always  occurred”)  of  breakage  or  malfunctions.  Five  of  1 1  survey  participants 

reported  problems  “sometimes”  or  more  with  only  two  helmet  components:  the  head-tracker  and 
the  CRTs. 

With  regard  to  the  head-tracker,  two  of  the  main  problems  identified  in  participant 
comments  were  that  it  malfunctioned  frequently  during  AGSS  sessions  and  that  it  did  not  keep  up 
with  required  scanning  rates.  One  participant  reported  that  certain  locations  in  the  AGSS  cabin 
were  problematic,  creating  diagonal  lines  in  the  visual  field. 

The  main  problems  noted  were  CRTs  performance  deterioration  over  time,  CRT  Mure 
rate,  and  perceptual  mismatches  from  using  old  and  new  CRTs  together.  For  example,  one 
participant  reported  that  sometimes  one  [CRT[  is  better  than  the  other  so  your  eyes  see  two 
different  things,  and  that  this  is  more  and  more  of  a  problem  as  time  goes  by.”  Another  individual 
stated  that  “the  new  ones  [CRTs]  bum  out  quickly,  creating  distracting  blotches  and  some 
disparity  between  the  two  eye  pieces.” 

Recommendations.  From  our  perspective,  it  is  critical  that  the  performance  of  the  head- 
tracker  and  CRTs  should,  at  the  very  least,  be  monitored  over  time.  In  addition,  the  CRTs  could 
be  replaced  more  frequently,  and  in  pairs,  to  prevent  large  visual  discrepancies  between  the  two 
eyes.  For  the  head-tracker,  more  recent  technology  with  fester  update  rates  should  perhaps  be 
explored.  These  recommendations,  however,  require  substantial  additional  funds  to  be  invested  in 
the  AGSS.  Until  the  addition  of  funding,  a  number  of  clever,  innovative  “work  arounds”  may 
offer  interim  solutions.  During  our  observations  of  training,  for  example,  one  particularly  adept 
instructor  recommended  that  students  use  an  “NVG  scan,”  which  is  a  slower  scan,  to 
accommodate  the  lag  between  the  head-tracker  and  student  scanning  rates.  In  this  way,  students 
can  maintain  “procedural  fidelity5’  without  creating  their  own  simulator-specific  “work  arounds” 
that  could  interfere  with  proper  skill  performance  in  the  aircraft.  Also,  if  the  CRTs  cannot  be 
replaced  more  frequently,  students  should  be  warned  about  possible  visual  discrepancies.  From 
our  experiences  in  other  training  environments,  students  are  much  more  receptive  to  new  training 
technologies  when  both  the  virtues  and  limitations  are  discussed  up  front.  ° 

Helmet  Comfort  and  Fit 

Comments.  The  frequency  of  problems  associated  with  ten  aspects  of  the  helmet  comfort 
and  fit  were  rated,  again  from  “never”  to  “always.”  Three  aspects  are  worth  discussing  because 
of  the  comments  associated  with  each.  They  are  the  helmet  pads,  the  occurrence  of  hot  spots 
and  the  chmstrap.  These  are  all  related  to  the  results  from  the  Kolmogorov-Smimov  analysis 
which  indicated  that  helmet  fit  is  only  “acceptable.” 

The  helmet  pads  were  frequently  cited  in  participant  comments  as  being  inadequate,  with 
their  method  of  attachment  to  the  helmet  singled  out  most  often.  Participants  noted  that  the 
velcro  tape  comes  off  inside  the  helmet,  and  that  the  pads  unglue  and  rip.  Others  reported  that 
the  pads  are  hard  and  uncomfortable.  In  responses  to  other  queries  about  poor  helmet  fit  or 
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helmet  slippage,  several  participants  also  suggested  that  the  helmet  pads  were  the  primary  culprit. 
Indeed,  one  participant  stated  directly  that  the  reason  the  helmet  fit  poorly  was  due  to  hard  pads. 
Despite  the  frequent  mention  of  helmet  pad  problems,  only  two  participants  reported  problems 
with  the  comfort  and  fit  of  the  helmet  pads  “sometimes”  or  more. 

The  second  aspect  of  note  was  the  report  by  four  survey  participants  of  hot  spots  when 
using  the  AGSS  helmet  even  after  short  (e.g.,  30  min)  AGSS  sessions.  The  helmet  pads  were 
again  cited  as  one  reason  for  the  hot  spots.  However,  several  respondents,  including  those  who 
did  not  rate  hot  spots  as  a  problem,  singled  out  the  helmet’s  method  of  use  as  a  key  factor.  They 
suggested  that  if  the  appropriate  adjustments  and  sizing  were  made,  hot  spots  could  be  avoided, 
such  that  “with  proper  pads  and  proper  adjustment  these  [hot  spots]  didn’t  occur.” 

Although  only  two  participants  reported  major  problems  with  the  chin  strap,  it  is  worth 
mentioning  because  of  its  relationship  to  poor  helmet  fit.  Participants  proclaimed:  “it’s 
uncomfortable,”  “it  cuts  into  your  skin,”  “no  real  padding,”  or  “needs  a  different  kind  of 
padding.”  They  also  suggested  that  the  helmet’s  method  of  use  influenced  their  ratings.  For 
example,  one  participant  said,  ‘If  the  chinstrap  is  upside  down  with  the  fabric  touching  your  face, 
this  is  a  problem,  and  this  can  occur  if  it  [the  helmet]  is  put  on  improperly  or  moves  during 
training.” 

Recommendations.  Short  of  acquiring  new  helmets  or  integrating  the  CRTs  and  head- 
tracker  into  the  same  helmets  worn  in  the  aircraft,  all  of  the  above  complaints  (helmet  padding, 
hot  spots,  and  chin  strap  discomfort)  could  be  remedied  by  improving  the  quality  of  the  pads  and 
their  method  of  attachment  to  the  helmet.  The  helmet  fitting  and  adjustment  process  also  needs  to 
be  monitored  closely.  Moreover,  as  the  AGSS  is  incorporated  into  MQ  and  Annual  Refresher 
training,  more  standardized  procedures  should  be  established  to  help  alleviate  problems  and/or 
discomfort  that  arise  from  improper  fit  helmet. 

Helmet  Electronic  and  Communication  Cables 

Comments.  Two  questions  concerning  the  helmet  cables  are  notable  because  of  the 
comments  associated  with  each.  First,  four  participants  reported  that  the  current  cable  length 
limited  their  amount  of  normal  movement  around  the  aircraft.  Respondents  qualified  their 
answers,  stating  that  for  the  subset  of  tasks  performed  in  the  AGSS,  their  movements  were  not 
hindered.  One  participant  explained  that  “cable  length  is  not  too  much  of  a  problem  because  of 
what  is  done  in  the  AGSS,  but  amount  of  movement  is  less  than  in  the  aircraft.  Tasks  requiring 
movement  are  things  like  switches,  weapon  malfunctions,  and  getting  equipment.”  Some  cited 
particular  stations  or  configurations  (methods  of  use)  as  more  problematic  than  others.  For 
example,  one  participant  reported  that  the  range  of  motion  in  the  tail  position  compared  to  the 
aircraft  was  reduced  in  the  AGSS  as  a  result  of  the  cable  length.  The  MH-60G  crewmember 
interviewed  also  noted  that  when  the  AGSS  was  in  the  MH-60G  configuration,  the  range  of 
motion  from  seat  to  door  in  the  left  and  right  positions  was  less  than  the  aircraft. 

Second,  three  participants  reported  qualified  problems  with  the  amount  of  pressure  they 
felt  from  the  cables  tugging  at  the  backs  of  their  head  when  performing  tasks  in  the  AGSS.  Those 
who  encountered  problems  with  this  feature  stated  various  caveats,  such  as,  problems  only 
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occurred  during  extreme  movements  or  that  the  cables  were  distracting  but  they  would  probably 
get  used  to  it.  However,  one  participant  did  report  that  he  felt  like  the  cables  “kept  getting  caught 
on  something,”  and  that  he  was  forced  to  stop  what  he  was  doing  in  the  AGSS  and  unhook  them 
from  his  back  two  or  three  times  during  training. 

Recommendations.  Due  to  the  small  number  of  respondents  reporting  cable  length 
problems  associated  with  task  performance  or  discomfort,  there  is  some  question  as  to  whether 
the  cables  really  are  a  problem.  Considering  the  participants’  comments,  it  seems  that  if  only  the 
appropriate  AGSS-supported  tasks  are  performed,  the  cable  length  is  satisfactory.  However, 
further  consideration  of  cable  length  for  the  tail  position  seems  warranted.  Given  some  of  the 
specific  comments,  longer  cables  may  be  necessary  for  the  tail  position  to  perform  his/her  required 
tasks.  In  addition,  a  few  respondents  did  suggest  that  a  means  of  tethering  the  various  cables  up 
off  their  shoulders  may  be  beneficial  for  the  times  when  the  cables  did  get  caught,  especially  for 
the  MH-60G  configuration,  where  the  crew  seats  are  used. 

Helmet  Donning  and  Adjustment 


Comments.  Included  in  the  helmet  subsection  of  the  survey  were  questions  that  asked 
participants  to  estimate  the  number  of  minutes  required  to  put  on  and  adjust  the  helmet  during 
their  first  and  subsequent  encounters  with  the  AGSS.  Participants  also  indicated  whether 
readjustments  were  necessary  during  training  sessions.  Average  donning  time  for  the  first  use  was 
11.1  min;  the  average  for  subsequent  sessions  was  3.2  min.  The  reduction  in  donning  time  was 
statistically  significant  (t  =  4.15,  p  <  .01,  df  =  9,  with  one  participant  excluded  because  he  only 
wore  the  helmet  once).  In  addition,  three  participants  reported  the  need  to  readjust  the  helmet 
“sometimes”  or  more  during  AGSS  sessions.  (Five  additional  participants  noted  problems 
“rarely”  and  some  of  their  comments  are  included  in  the  table  below.)  Three  categories  of 
adjustment  problems  appeared  in  participants’  comments.  These  involved  issues  of  alignment,  the 
air  pump,  and  familiarity.  Table  1 1  summarizes  the  participants  verbatim  responses. 


Table  1 1 .  Survey  Participant  V erbatim  Responses  Regarding  Helmet  Adjustments. 


Category 

Participant  Responses 

Alignment 

=>  Had  problems  with  the  alignment  process. 

=>  Figuring  out  what  alignment  means  seems  to  be  the  biggest 
hurdle  -when  first  setting  up  the  helmet 
=>  It  seemed  oft  but  I  didn’t  want  to  adjust  it  because  I  didn’t 
want  to  screw  with  the  alignment. 

Air  Pump 

=>  Miscellaneous  problems  with  the  air  pump 
=>  Adjusting  pump.  Normal  head  movements  changes  how 
helmet  feels,  makes  you  adjust  comfort  level,  often  too  tight 
Found  myself  adjusting  it  every  5  minutes  or  so. 

=>  It  was  too  tight  and  I  had  to  release  some  air  pressure. 

=>  Pumped  it  up  too  tight  initially  and  let  air  out  as  flight 
progressed,  ended  up  releasing  most  of  the  air. 

=>  Two  hot  spots  in  the  back  were  relieved  by  letting  out  air 

Familiarity 

Once  familiar  with  its  operation,  it  is  okay.  Operator  error 
caused  some  initial  problems. 

=>  Had  to  shift  it  a  bit  just  to  get  it  comfortable.  If  it  is  not  set 
up  correctly,  you  will  have  serious  problems. 
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Recommendations.  It  is  not  surprising  that  the  helmet  adjustment  process  takes  longer 
during  the  first  use,  nor  is  it  unexpected  that  readjustments  are  required  during  training  sessions 
Nevertheless,  AGSS  curriculum  developers  need  to  be  aware  of  these  time  requirements  so  tha 
tasks  and  objectives  can  be  tailored  to  the  allotted  time  periods.  As  mentioned  previously,  the 
proper  adjustment  and  alignment  of  the  AGSS  visual  input  system  (e.g.,  CRTs)  is  critical.  Here 
again,  proper  alignment  procedures  are  recognized  as  a  primary  factor  for  reducing  readji^tments 
during  AGSS  flights.  Two  main  recommendations  are  made  in  this  regard.  First,  the  58  iKbb 
should  attempt  to  standardize  and  document  the  fitting  and  aligning  process  so  that  when 
knowledgeable  instructors  and  maintenance  support  relocate,  their  knowledge  does  not  leave  with 
them.  Second,  the  squadron  should  provide  additional  (or  separate)  time  prior  to  training  to 
familiarize  trainees  with  the  helmet,  VR  technology  and  terminology,  and  alignment  procedures. 


Trainee  Stations 

Seven  individual  trainee  station  components  (gun  feel,  gun  set-up,  mobility  in  station,  VR 
aircraft  interior,  VR  other  stations,  crew  seats  and  crew  harnesses)  were  assessed  in  our  survey. 
Five  are  discussed  below  based  on  the  Kolmogorov-Smimov  analysis  and  participant  comments. 


Gun  Set-IJn  and  Feel 

Comments.  Seven  participants  rated  gun  set-up  as  a  “3”  or  less.  The  primary  problems 
identified  concerned  gun  alignment  and  adjusting  the  gun  barrels  properly  within  the  AGSS. 
Similarly,  six  participants  rated  the  virtual  mapping  of  the  surrounding  gunner  stations  as  a  “3”  or 
less.  The  main  criticism  involved  the  virtual  mapping  of  the  other  players  in  the  aircraft 
environment.  Specifically,  there  is  an  FE  “mapped”  into  the  middle  seat  when  one  looks  forward 
in  the  AGSS  virtual  environment,  but  no  other  members  of  the  cockpit  nor  the  backend  are 
“mapped.”  Several  individuals  commented  on  this  selective  mapping.  Some  participants 
suggested  mapping  players  into  all  crew  positions  (i.e.,  model  a  pilot  and  copilot  in  the  other 
cockpit  seats),  while  others  advocated  eliminating  the  mapping  of  virtual  players  altogether. 

One  participant  provided  two  important  insights  regarding  the  general  fidelity  of  the 
trainee  stations.  First,  he  noted  that  he  was  more  likely  to  downgrade  the  fidelity  of  the  other 
stations  (vs  his  own)  because  he  could  more  easily  “nitpick”  what  should  and  should  not  be  there 
as  he  looked  around.  Secondly,  he  noted  that  if  he  was  looking  at  the  other  stations  and  not  out 
his  window,  then  he  was  not  doing  his  job.  The  tradeoffs  between  selective  fidelity  and  ‘Time  on 
task”  may  need  to  be  explored  further  as  the  AGSS  becomes  more  fully  integrated  into  the 
training  curriculum. 

Recommendations.  First,  the  difficulties  with  gun  barrel  alignment  can  be  reduced  by 
standardizing  and  better  conveying  the  general  alignment  process.  In  terms  of  mapping  other 
players,  the  two  choices  seem  to  be:  (a)  either  map  all  the  members  of  the  cockpit  in  the  AGSS 
(which  may  not  be  possible  given  current  IG  modeling  limitations),  or  (b)  map  none  of  them. 
Here,  again  the  tradeoffs  of  selective  fidelity  should  be  explored.  For  example,  mapping  aft  of  the 
cockpit  players  may  increase  user-acceptance  of  the  device  which  may  in  turn  have  a  beneficial 
effect  on  the  training  effectiveness  of  the  device. 
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Mobility.  Seats,  and  Harnesses 


Comments.  Next,  six  participants  rated  their  mobility  (already  briefly  mentioned  in 
reference  to  the  helmet  cables)  at  the  AGSS  gunner  stations  as  a  “3”  or  less.  The  primary 
criticism  was  that  there  is  less  mobility  in  the  AGSS  than  in  the  aircraft.  Several  different  reasons 
were  cited,  including  cable  length,  inhibitions  about  having  an  expensive  helmet  on  one’s  head, 
and  differences  between  the  virtual  and  physical  world.  Regarding  the  latter,  it  appears  as  if  one 
can  put  one’s  head  out  of  the  AGSS  windows  in  the  virtual  aircraft  although  the  simulator’s 
structure  does  not  allow  this.  However,  most  respondents  reiterated  that  given  the  tasks 
performed  in  the  AGSS,  reduced  mobility  was  not  that  much  of  a  problem. 

Due  to  the  limited  use  of  the  AGSS  in  the  MH-60G  configuration,  only  five  participants 
were  qualified  to  rate  the  functionality  and  fidelity  of  the  AGSS  seats.  Yet,  four  of  them  rated  the 
seats  as  a  “3”  or  less.  The  primary  problem  was  that  while  the  seats  met  the  design  specifications 
for  the  AGSS,  they  were  still  not  comparable  to  the  aircraft.  As  a  result,  participants  expressed 
concern  about  their  durability. 

The  harnesses  were  also  only  rated  by  five  participants,  and  again  four  of  them  rated  the 
harnesses  as  a  “3”  or  less.  Two  main  points  arose  from  individual  comments:  First,  the  harnesses 
seemed  to  be  of  poor  quality.  Second,  they  simply  were  not  used  muck  Respondents  noted  that 
while  the  harnesses  were  a  part  of  the  critical  design  review  (CDR)  requirements  for  the  AGSS, 
their  purpose  within  the  AGSS  was  unclear. 

Recommendations.  If  the  harnesses  and  seats  are  going  to  receive  regular  use,  the 
primary  recommendation  is  to  improve  their  quality  to  support  AGSS  training  throughput. 

IOS 


Thirteen  individual  features  of  the  IOS  were  rated  on  training  utility.  Six  are  discussed 
below  based  on  the  Kolmogorov-Smimov  analysis  and  participant  comments. 

Screen  Layout 

Comments.  Four  out  of  eight  participants  rated  the  student  screen  layout  at  the  IOS  as  a 
“3”  or  less,  and  three  out  of  eight  rated  the  general  layout  of  the  IOS  as  a  “3”  or  less.  The 
primary  critique  was  that,  while  the  current  layout  was  functional,  better  layouts  were  possible. 
One  individual  noted  that  due  to  the  height  and  location  of  the  instructor  information  output 
screen  (see  Figure  6),  he  had  to  get  out  of  his  seat  to  make  some  control  inputs. 

Recommendations.  There  are  two  main  recommendations  for  improving  the  IOS  layout 
and  making  the  IOS  more  “user  friendly.”  First,  if  the  IOS  layout  remains  as  shown  in  Figure  6, 
AGSS  training  would  be  greatly  enhanced  by  using  two  instructors  for  the  training  sessions.  One 
instructor  would  be  responsible  for  running  the  AGSS  and  the  other  for  observing,  instructing, 
and  role  playing.  (We  make  this  same  recommendation  later,  in  response  to  instructor-reported 
levels  of  task  saturation  at  the  IOS.)  Second,  if  additional  funds  are  available  or  the  AGSS  is 
moved,  a  more  ergonomic  arrangement  of  the  screens  was  suggested  by  a  survey  participant,  and 
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that  view  is  endorsed  here.  The  proposed  layout  (see  Figure  8)  would  enable  instructors  to  better 
divide  their  attention  between  IOS  inputs  and  instruction,  while  also  eliminating  the  instructor’s 
disruptive  requirement  to  get  out  of  the  seat  to  make  inputs  during  training. 
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Figure  8.  Recommended  Improved  Layout  of  IOS  Screens. 


Space  Ball 

Comments.  The  most  unanimously  agreed-upon  poor  feature  of  the  IOS  was  the  space 
ball  for  flying  ownship  during  stand-alone  AGSS  training.  Nine  out  of  nine  participants  rated  the 
space  ball  as  a  “3”  or  less,  and  it  was  singled  out  by  the  Kolmogorov-Smimov  analysis  as  having 
significantly  negative  ratings.  This  result  is  in  contrast  to  only  three  of  nine  who  rated  the  space 
ball  as  a  “3”  or  less  for  flying  moving  models  and  the  “acceptable”  result  of  the  Kolmogorov- 
Smimov  analysis  for  space  ball  flying  of  moving  models.  This  is  a  prime  example  of  how  the 
method  of  use  of  a  particular  AGSS  feature  has  a  great  influence  on  its  judged  effectiveness.  The 
central  reason  given  for  the  overwhelming  displeasure  with  the  space  ball  for  flying  the  ownship 
was  the  poor  software  modeling  of  aircraft  aerodynamics  which  manifests  itself  as  a  tremendous 
difficulty  for  maintaining  proper  flight  control  using  the  space  ball.  The  following  excerpt  is 
representative  of  the  space  ball  critiques  we  recorded,  and  is  illustrative  of  the  problems  and 
consequences  of  using  the  space  ball  for  ownship  flying  during  training: 

The  big  thing  with  space  ball  ownship  flying  is  the  MH-53J  WST  code  that  is  used 
for  the  software  modeling.  This  causes  major  problems  if  you  are  forced  to  fly  with 
the  space  ball  during  stand-alone  training  with  students.  It  is  impossible  to  interact 
with  students  as  you  should.  You  [also]  end  up  using  the  slew  keys  more  often  than 
you  should  which  gives  students  artificial  cues.  .  .  Do  anything  to  stay  away  from 
space  ball  flights.  Get  rid  of  space  ball  or  make  it  fly  more  like  a  generic  model. 

Recommendations.  With  such  unanimous  displeasure  associated  with  flying  the  ownship 
using  the  space  ball,  we  strongly  recommend  that  it  either  be  changed  or  its  use  limited  during 
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training.  Several  participants  suggested  simply  replacing  the  space  ball  with  a  joystick,  enabling 
quicker  input  adjustments  and  accommodations  due  to  poor  modeling  of  aircraft  aerodynamics 
However,  at  the  AGSS’s  current  level  of  development,  this  would  require  extensive  and  costly 
changes  to  die  software  (e.g.,  changing  drivers).  Another  less  optimal  suggestion  is  to  conduct 
AGSS  training  with  minimal  use  of  the  space  ball,  either  in  the  integrated  mode — when  the  MH- 
60G  or  the  MH-53J  is  the  host  cockpit — or  using  the  record-replay  function  to  run  through 
prerecorded  missions.  This  latter  approach  is,  in  fact,  often  employed  during  current  AGSS 
training.  Among  its  advantages,  this  method  greatly  increases  the  instructor’s  ability  to:  (a)  focus 
on  the  students  in  the  AGSS,  (b)  provide  timely  feedback,  (c)  use  the  performance  monitoring 
pages,  and  (d)  introduce  moving  models  into  the  scenario.  However,  the  interactive  nature  and 
realism  of  the  flights  are  reduced  because  directional  calls  (e.g.,  break  left)  cannot  be  responded  to 
unless  it  has  been  prerecorded.  Use  of  the  integrated  mode,  on  the  other  hand,  will  solve  many  of 
the  problems  associated  with  the  space  ball  for  ownship  flying.  Yet  this  mode  is  still  viewed  as 
less  than  optimal  since  it  is  not  presently  integrated  and  (three)  stand-alone  AGSS  rides  are  still  a 
required  part  of  the  AG  Mission  Qualification  training. 

Second  Instructor  Position 

Comments.  There  was  extensive  agreement  among  participants  on  lack  of  utility  of  the 
second  instructor  position  at  the  IOS.  Six  out  of  seven  participants  rated  this  IOS  feature  as  “3”  or 
less.  Comments  focused  on  inability  of  instructors  at  this  position  to  communicate  directly  with 
students  in  the  AGSS.  Echoing  the  thoughts  of  the  other  five  displeased  participants,  one 
individual  succinctly  said,  “You  have  the  capability  to  hear,  but  you  can’t  talk.  This  is  dumb 
Provide  this  capability.” 

Recommendations.  In  order  to  enable  instruction  and  accommodate  some  of  our  other 
recommendations,  the  second  instructor  position  must  be  provided  the  capability  to  communicate 
directly  with  the  students  in  the  AGSS. 


Individual  IOS  Pages 

Comments.  Other  IOS  features  rated  as  “3”  or  less  by  a  number  of  survey  participants 
were  three  IOS  pages:  flare  monitoring,  threat  status,  and  the  ground  track  map.  Five  of  seven 
participants  rated  the  flare  performance  monitoring  page  as  “3”  or  less  and  four  out  of  seven 
participants  similarly  rated  die  threat  status  page.  For  both  pages,  however,  the  comments 
primarily  addressed  die  fact  that  they  “just  weren’t  used  much,”  suggesting  a  degree  of  skepticism 
regarding  content  and  utility  of  these  pages.  Five  of  eight  participants,  however,  rated  the  ground 

track  map  page  as  3  or  less.  The  primary  criticism  of  the  ground  track  map  page  was  its  lack  of 
waypoint  representation. 

Recommendations.  A  solution  to  the  absence  of  waypoints  on  the  ground  track  map 
Paf-s-ply  t0  Pr0Ylde  the111  as  m  some  of  the  other  simulators  at  the  58  TRSS.  However  the 
AGSS  lacks  a  navigation  system,  and  without  this  capability,  it  is  not  possible  to  input  waypoints. 
The  costs  to  implement  a  one-of-a-kind  navigation  imitator  on  the  AGSS  would  undoubtedly 
override  any  debatable  benefits  on  a  device  such  as  the  AGSS.  Nevertheless,  there  is  at  least  one 
potential  solution  given  the  current  AGSS  design  specifications.  One  survey  participant  offered 
the  clever  suggestion  of  setting  up  inactive  (or  active)  threats  to  mimic  waypoints  on  the  ground 
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track  map  page.  Threats  do  appear  on  the  ground  track  map  and  could  be  used  to  help  navigation 
calls,  especially  for  the  FEs. 


Fvternal  Environment 

Comments.  Several  questions  were  designed  to  tap  into  the  existence  of  potential 
problems  associated  with  locating  the  IOS  outside  the  motion  base  of  the  AGSS  (unlike  other 
simulators  at  the  58  TRSS).  Question  topics  included  noise  level,  visual  distractions,  and  the 
instructor  level  of  task  saturation  due  to  the  inability  to  see  students  and  monitor  screens  for  both 
student  performance  and  aircraft  performance.  The  only  question  having  a  large  proportion  of 
participants  rating  it  as  a  problem  “sometimes”  or  more  was  the  level  of  instructor  task  saturation 
experienced  at  the  IOS.  Task  saturation  was  noted  by  participants  whether  there  was  one  student 
(5  of  8),  two  students  (5  of  8),  or  three  students  (4  of  6)  in  the  AGSS. 

The  mean  levels  of  instructor  task  saturation  experienced  with  one,  two,  and  three  students 
is  shown  in  Figure  9.  As  can  be  seen,  there  is  a  linear  relationship  between  the  reported  level  of 
task  saturation  and  the  number  of  students  in  the  AGSS.  Thus,  as  the  number  of  students  in  the 
AGSS  goes  up,  so  does  reported  level  of  task  saturation.  Although  the  comparisons  between  the 
means  were  not  statistically  significant,  participant  comments  suggested  that  the  demands  of  flying 
ownship,  laying  down  threats,  and  monitoring  students  can  wreak  havoc  on  the  level  of  training 
that  instructors  are  able  to  provide  students  in  the  AGSS.  Figure  9  also  shows  that  the  severity  of 
reported  task  saturation  increases  when  the  one  SME,  who  had  a  large  role  in  the  development  of 
AGSS  IOS  and  over  eight  years  experience  as  a  simulator  instructor/operator,  is  removed  from  the 
calculations  (i.e.,  the  right  bar  graph  of  each  pair  in  Figure  9). 


Figure  9.  Reported  Levels  of  Instructor  Task  Saturation  with  One,  Two,  and  Three  Students  in  the 

AGSS. 
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Recommendations.  There  are  at  least  two  potential  solutions  for  reducing  the  level  of 
instructor-reported  task  saturation:  limit  the  number  of  students  who  are  trained  concurrently  in 
the  AGSS  or  add  a  second  instructor  at  the  IOS  (as  already  mentioned).  We  view  the  first 
solution  as  inadequate.  Although  the  instructor’s  ability  to  provide  feedback  and  observe  training 
is  increased,  the  level  of  realism  of  training  and  the  number  of  skills  that  can  be  practiced  (e.g.. 
Left,  Right,  Tail  call  responsibilities  or  crew  communication)  are  significantly  reduced.  The 
second  solution  is  really  the  best,  provided  the  second  instructor  position  is  enhanced  to  include 
the  ability  to  directly  communicate  with  students  in  the  AGSS.  Two  instructors  at  the  AGSS 
would  allow  increased  realism,  better  feedback,  more  role  playing,  and  better  use  of  the  available 
AGSS  capabilities. 

Although  administered  as  its  own  section  during  the  interviews,  questions  concerning 
instructor/student  interaction  and  the  communication  system  are  logically  related  to  the  above 
issues,  so  its  results  are  noted  here.  Several  yes/no  questions  about  instructor  and  student  ability 
to  hear  one  another  were  asked.  None  of  the  participants  reported  significant  problems  with 
either  hearing  or  understanding  the  students  while  at  the  IOS,  nor  were  participants  unable  to  hear 
or  understand  one  another  or  the  instructor  while  in  the  AGSS. 

Weapons  Simulation 

Based  on  the  Kolmogorov-Smimov  analysis  and  participant  comments,  six  individual  gun 
features  are  discussed  below.  The  overall  assessment  of  the  weapons  simulation  is  also  discussed 
due  to  the  number  of  participants  rating  it  as  a  “3”  or  less. 

Modeling  Fidelity 

Comments.  Two  related  items,  the  ballistic  model  and  tracer  fidelity,  both  had  a  large 
number  of  individuals  rating  them  as  a  “3”  or  less:  eight  individuals  for  tracer  fidelity  and  six  for 
the  ballistic  modeL  Although  the  specific  comments  for  each  element  were  different,  a  common 
problem  was  inadequate  mapping  of  the  projectile  jump,  or  lead  and  lag,  of  the  bullet’s  path.  In 
particular,  the  problem  arises  from  using  F-16  (an  aircraft  with  guns  mounted  on  the  nose) 
software  for  modeling  the  bullet  path.  In  the  MH-60G  or  MH-53J,  two  of  the  guns  are  mounted 
on  the  sides  of  the  aircraft  leading  to  different  aerodynamics  on  the  bullet.  This  requires  different 
aiming  procedures  since  a  bullet  fired  from  the  side  window  of  a  helicopter  does  not  travel  in  a 
straight  line  as  it  would  if  it  were  fired  off  the  nose  (or  tail).  Although  most  comments  focused  on 
this  issue,  and  how  the  AGSS  would  be  better  if  this  were  incorporated,  one  individual  said:  “The 
bullets  go  straight  and  this  does  not  teach  anything.  You  just  can’t  replicate  real  guns.”  Other 
critiques  about  the  tracer  path  and  the  ballistic  model  concerned  the  number  of  tracers  per  rounds 
fired,  the  different  cues  in  the  AGSS  (compared  to  the  aircraft),  the  inability  to  see  tracers  leaving 
the  gun,  and  the  existence  of  jerky  visuals  during  target  engagement. 

Recommendations.  The  primary  recommendation  regarding  the  ballistic  model  is  to 
improve  it  by  incorporating  lead  and  lag  effects.  Again,  however,  such  a  change  requires 
extensive  modifications  to  software  and  additional  AGSS  funding.  Alternatively,  if  this  is  not  an 
option,  students  may  be  instructed  about  the  differences  between  the  AGSS  and  the  aircraft.  The 
AGSS  could  be  used  as  a  potent  illustrator  to  contrast  the  bullet’s  path  in  the  aircraft  versus  the 
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simulator.  Due  to  current  IG  modeling  constraints,  the  limited  number  of  tracers  per  rounds  fired 
may  also  require  additional  “coaching.” 

Alifnment 

Comments.  Another  aspect  of  weapons  simulation  receiving  almost  unanimous 
displeasure  was  the  alignment  of  the  virtual  gun  with  the  physical  gun  in  the  data  base,  with  nine 
participants  rating  it  as  a  “3”  or  less.  The  two  main  criticisms  were  the  (a)  unfamiliar  and 
unstandardized  alignment  process  and  (b)  apparent  degradation  of  the  alignment  during  training. 

Similarly,  the  alignment  of  the  gun  on  its  pivot  point  was  rated  as  a  “3”  or  less  by  eight 
participants.  The  main  issues  cited  were  that  the  gun  “floats,”  the  alignment  process  itself  is  poor, 
and  it  is  necessary  to  make  readjustments  during  flight.  Many  participants  suggested  that  this 
problem  was  strongly  related  (again)  to  lack  of  understanding  and  definition  of  the  appropriate 
procedures  for  aligning  the  gun  position  in  the  data  base. 

Recommendations.  Again,  as  mentioned,  we  recommend  improving  and  standardizing 
the  alignment  procedures  to  alleviate  many  of  these  problems.  We  would  also  recommend 
improving  the  modeling  of  the  gun  in  the  data  base.  (Note:  Since  the  survey  was  conducted,  the 
software  modeling  of  the  gun  position  has,  indeed,  been  improved  and  there  is  no  longer  a  gap 
between  the  gun  and  the  pivot  point  [i.e.,  the  “floating  gun”]). 

Targeting 

Comments.  Target  acquisition  was  also  surveyed  as  part  of  AGSS  weapons  simulation, 
with  seven  participants  rating  it  as  a  “3”  or  less.  The  primary  comments  were  related  to  poor 
characteristics  of  the  visual  system,  as  exemplified  by:  “really  due  to  poor  visuals,”  “associated 
with  visuals,”  or  “can’t  see  targets  until  they  are  very  close,  does  not  equal  reality,  or  even  come 
close.” 


Four  survey  participants  rated  airstream  effects  as  a  “3”  or  less.  The  primary  comments  in 
this  area  were  related  to  aircraft  and  simulator  discrepancies  (e.g.,  control  loading)  during  AGSS 
development.  (Note:  Since  the  survey  was  conducted,  this  problem  has  also  been  remedied.) 

Recommendations.  The  primary  recommendation  in  terms  of  improving  the  target 
acquisition  capability  of  the  AGSS  is  to  improve  the  quality  of  the  AGSS  visual  system.  This  will 
be  discussed  in  further  detail  in  the  next  section. 

Finally,  seven  participants  rated  the  overall  weapons  simulation  of  the  AGSS  as  a  “3”  or 
less.  However,  the  Kolmogorov-Smimov  analysis  indicated  that  weapons  simulation  was  rated 
significantly  positive.  Participants’  comments  support  this  apparent  contradiction.  The 
overwhelming  sentiment  expressed  in  terms  of  weapons  simulation  was  that  participants  were  very 
enamored  with  the  new  capabilities  of  the  AGSS — indicated  by  high  participant  ratings.  They 
were,  however,  disappointed  in  several  key  features  (the  ballistic  modeling,  visual  system,  and 
alignment  process)  and  this  is  clearly  indicated  in  their  comments.  In  essence,  this  combination  of 
results  is  typical  of  any  new  technology  assessment — participants  recognized  the  AGSS’s  training 
potential,  but  were  also  well  are  of  its  current  shortcomings. 
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Visual  System 

Comments.  As  indicated  by  the  Kolmogorov-Smimov  analysis,  various  features  of  the 
visual  system  were  rated  “acceptable”  or  lower.  We  discuss  several  of  these  below,  as  well  as  any 
previously  unmentioned  differences  between  NVG  night  simulation  and  day  simulation  that 
participants’  noted. 

Eight  participants  rated  object  detail  and  visual  resolution  as  a  “3”  or  less.  Comments 
highlighted  the  fact  that  the  level  of  detail  did  not  match  the  real  world;  this  was  particularly 
obvious  for  targets.  As  a  behavioral  consequence  of  this  lack  of  detail,  these  eight  participants 
also  rated  target  or  object  detectability  as  a  “3”  or  less.  Comments  focused  on  lack  of  detail  and 
the  inability  to  make  friend  versus  foe  distinctions  unless  objects  were  at  an  unrealistically  close 
range.  Particular  object  representations  were  also  rated,  including  a  747,  explosions,  a  Hind-D 
helicopter,  jeeps,  roads,  runways,  surface-to-air  missile  (SAM)  sites,  trees,  and  trucks.  Not 
surprisingly,  given  the  overall  ratings  of  object  resolution,  seven  of  these  objects  were  rated  low 
by  a  large  proportion  of  survey  participants.  The  primary  reason  for  the  low  ratings  in  each  case 
was  insufficient  detail.  Table  12  presents  a  summary  of  the  data  for  each  object.  The  shaded  rows 
(runways  and  explosions)  indicate  the  objects  with  the  fewest  number  of  low  ratings. 

Table  12.  Specific  Object  Ratings  and  Representative  Comments. 


Object 

Observed  Number  of 
Ratings  of  3  or  Less 

Representative  Participant  Comment 

747 

5  of  9 

None  noted. 

Hf  igjg 

!  -It  :  -I; 

Helicopter 

(Hind-D) 

5  of  11 

=>  Needs  more  detail,  a  higher  texture  model. 

=>  Lacked  clarity,  couldn’t  tell  if  it  was  airborne. 

Jeep 

6  of  10 

=>  As  targets  okay,  but  if  you  had  to  tell  between  good  guys 
and  bad  guys,  you  couldn’t  do  it. 

=>  Was  told  it  was  there,  but  didn’t  recognize  it. 

Roads 

8  of  11 

=>  Couldn’t  distinguish  between  roads  and  rivers. 

=>  Just  brown  lines. 

<  ' 

tr 

&  ''  "* 

j  ii  * /  i-  a  j  *  .  ....  '  y  '  y 

f  iL  r  '  . 

;y  ,y  -1 

Li! 

SAM  Sites 

6  of  9 

Animation  needs  to  be  upgraded. 

=>  Could  tell  up  close  (400  ft),  but  farther  away,  problems. 

Trees 

9  of  11 

=>  Need  work,  a  lot  of  variability. 

=>  Other  trees  in  other  visual  systems  are  better.  Trees.are  flat. 

Truck 

6  of  10 

=>  Not  enough  detail. 

=>  Could  have  been  a  white  blur. 
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Similarly,  we  also  asked  about  image  resolution  and  detail  level  of  the  entire  visual  scene, 
Six  of  1 1  participants  rated  this  feature  as  a  “3”  or  less.  Comments  included:  “could  be  more 
stuff  in  there  ”  “lack  of  detail,”  and  “lack  of  realism.” 

A  critical  element  of  any  visual  system  is  its  representation  of  depth.  We  asked  about  this 
feature  in  two  ways.  First,  we  asked  for  a  general  assessment  of  whether  the  visual  scene 
provided  meaningful  depth  information.  We  also  examined  the  behavioral  manifestation  of  this  in 
terms  of  participants’  distance  perception  abilities.  Five  participants  rated  the  portrayal  of 
meaningful  depth  in  the  visual  scene  as  a  “3”  or  less.  Accordingly,  participants’  judged  distance 
perception  as  poor,  with  six  participants  rating  it  as  a  “3”  or  less.  Participants  had  different  views 
concerning  the  primary  culprit.  Several  individuals  cited  the  above  problems  with  object 
resolution  as  the  key  cause-  Others  blamed  their  perception  difficulties  on  the  “flat  scene.”  Still 
others  qualified  their  comments,  saying  that  certain  distances  could  not  be  judged  but  others 
could.  For  example,  one  participant  said  that  judging  distance  from  the  aircraft  to  the  ground  was 
possible  in  the  AGSS,  but  other  aspects  of  distance  judgment  were  difficult.  The  variability  in 
responses  led  us  to  believe  that  perhaps  the  main  problem  with  this  visual  feature  was  certain 
inevitable  differences  between  the  aircraft  and  the  AGSS. 

Related  to  both  resolution  and  depth  of  the  visual  scene  is  visual  contrast-  Seven 
participants  rated  this  feature  as  a  problem.  Participants’  comments  were  fairly  specific, 
interpreting  their  inability  to  discern  visual  contrast  as  a  result  of  the  limited  detail  in  the  visual 
scene  and  the  lack  of  a  three-dimensional  scene  representation.  A  particularly  telling  comment 
was,  “My  focusing  is  impaired,  constantly  fighting  to  focus,  trying  to  grab  at  sharp  detail,  but  not 
much  there.”  One  participant,  however,  did  mention  that  visual  contrast  was  satisfactory  during 
the  daytime  scenes,  but  it  was  not  sufficient  during  NVG  night  scenes. 

Five  of  1 1  participants  rated  visual  brightness  as  a  “3”  or  less.  Like  the  above  feature,  and 
in  accord  with  our  framework  (Figure  2),  there  appeared  to  be  some  disparity  due  to  method  of 
use.  Specifically,  NVG  night  scenes  were  the  main  source  of  the  critical  comments,  e.g.,  “the 
virtual  NVG  scene  is  grayer  than  the  NVG  scene  in  the  aircraft  and  the  color  balance  seems  off” 

Visual  scene  clarity  and  image  clarity  were  also  judged  as  problematic  by  a  number  of  the 
participants  (7  and  5,  respectively),  Participants  claimed  that  although  the  system  was  good  in  its 
present  form,  there  was  room  for  marked  improvement. 

The  completeness  of  the  visual  scene  was  rated  as  a  “3”  or  less  by  six  participants-  They 
qualified  their  responses  to  reflect  discrepancies  between  a  properly  versus  a  poorly  functioning 
IG,  The  central  theme  was  that  when  the  IG  was  functioning  properly,  the  scene  did  not  get 
chopped  or  clipped  off.  But  when  it  was  not  working,  “it  would  do  some  weird  stuff.” 

Visual  scene  continuity  was  rated  as  a  problem  by  seven  participants,  but  the  comments 
were  not  qualified  by  the  status  of  the  IG.  Participants  were  primarily  concerned  about  the  update 
rate  of  the  IG,  maintaining  that  it  seemed  slow  and  that  occasionally  the  scene  had  to  catch  up 
with  their  scanning  rates. 
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Visual  alignment  ratings,  as  previously  reported,  were  unsatisfactory.  In  this  case,  the 
assessment  focused  on  the  visual/virtual  representation  of  the  physical  world,  with  six  participants 
rating  it  as  problematic.  Most  of  the  comments  identified  small  details  of  the  virtual  environment 
that  did  not  match  the  aircraft.  One  participant,  however,  said  “that  [the  AGSS]  was  more  two- 
dimensional  than  [he]  would  like  and  that  it  was  not  as  three-dimensional  as  VR  technology 
claims.” 


Image  color  was  rated  as  problematic  by  six  participants.  The  main  complaints  were  that 
the  colors  looked  “washed  out”  and  that  there  were  noticeable  differences  between  new  and  old 
CRTs.  Importantly,  AGSS  image  color  has  been  improved  since  the  survey,  but  the  issue  of  CRT 
breakage  and  decay  still  needs  to  be  addressed. 

Although  rated  significantly  positive  (see  Table  10)  by  our  survey  participants,  we  probed 
the  FOV  feature  further  because  of  its  importance  in  the  fulfillment  of  AG/S  tasks.  We  asked  two 
yes/no  questions  regarding  FOV  for  daytime  and  NVG  night  conditions.  Seven  of  1 1  participants 
reported  that  they  did  not  achieve  a  full  FOV  for  daytime  operations,  but  only  two  of  nine 
participants  reported  being  unable  to  achieve  a  full  FOV  for  NVG  night  operations.  This  likely 
reflects  the  fact  that  during  NVG  use,  one’s  peripheral  vision  is  already  reduced  and  the  AGSS 
visual  system  is  able  to  mimic  this.  The  primary  concern  during  daytime  operations  was  that  even 
with  their  understanding  of  the  visual  system’s  constraints  on  FOV,  participants  were  unable  to  see 
fully  forward  or  aft  because  they  could  not  “get  their  heads  out  of  the  windows”  as  many 
(particularly  FEs)  do  in  the  aircraft. 

Not  surprisingly,  seven  survey  participants  gave  low  overall  assessments  of  the  visual 
system.  The  specific  comments  seemed  to  reflect  the  visual  system  features  participants  were 
most  dissatisfied  with,  such  as  the  CRTs,  distance  cues,  visual  contrast,  etc.  But  despite  their 
apparent  dissatisfaction,  several  comments  suggested  that  participants  could  train  with  the  device 
as  is,  but  there  was  room  for  “a  lot  of  improvement.” 

Recommendations.  Participant  dissatisfaction  with  the  visual  system  was  quite  pervasive. 
As  a  main  component  of  the  AGSS  and  given  its  criticality  to  AG/S  tasks,  a  primary 
recommendation  is  to  explore  the  visual  system  in  further  detail,  to  gain  a  better  understanding  of 
the  many  problems  encountered  by  survey  participants  and  which  features  are  the  most  important 
to  the  fulfillment  of  AG/S  skills.  While  many  of  the  comments  focused  on  definite  inadequacies 
with  the  system  (e.g.,  washed  out  color  or  faulty  IG),  others  seemed  to  be  related  to 
misconceptions  about  the  capabilities  of  simulation  (e.g.,  discrepancies  between  real-world  image 
and  the  virtual  image).  With  a  substantial  percentage  of  the  users  (both  instructors  and  trainees)  of 
this  device  being  new  to  simulation,  it  may  be  that  many  of  their  critiques  have  more  to  do  with  a 
lack  of  knowledge  about  simulation  capabilities  rather  than  inadequate  device  capabilities  This 
would  suggest  that  additional  or  separate  familiarization  time  be  provided  where  the  device  and  its 
capabilities  are  briefed.  In  addition,  further  exploration  of  the  visual  system  would  distinguish 
areas  for  which  there  are  real  problems  with  the  hardware/software  from  those  which  are  user- 
centered.  Based  on  the  survey  and  the  skills  the  AGSS  is  designed  to  train,  three  visual  system 
features  require  immediate  attention:  object  detail,  color  quality,  and  visual  continuity  (or  update 
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We  also  recommend  improving  the  FOV,  but  not  by  purchasing  a  new  visual  display  The 
main  atticism  of  the  FOV  was  the  inability  to  see  fully  forward  or  aft  in  the  AGSS.  The  primary 
reason  for  this  was  not  the  limited  range  of  the  HMD's  FOV,  but  instead,  it  was  the  trainee? 

their  FOV°  ^  ?R  helmets  °ut  the  AGSS  side  windows  and  use  head  movements  to  expand 
their  FOV,  as  presently  done  in  the  aircraft.  (However,  there  is  some  disagreement  among  SMEs 

as  to  the  actual  frequency  of  this  practice.)  One  solution  would  be  to  enlarge  the  AGSS  windows 
to  accommodate  the  VR  helmets.  While  this  would  reduce  the  AGSS’  physical  fidelity  with  the 
aircraft,  its  functional  fidelity  would  be  much  enhanced. 

Motion 


The  miscellaneous  section  of  the  survey  contained  several  questions  about  the  motion 
system.  Two  questions  and  their  responses  offered  some  insight  in  this  regard  First  six 
participants  reported  that  the  AGSS  did  not  fly  like  the  aircraft.  Yet,  eight  participants  reported 
bemg  satisfied  with  the  three-degree-of-freedom  motion  system.  Combined,  these  two  results  led 
to  the  tentative  conclusion  that  the  limited  motion  base  was  not  the  primary  reason  why 
participants  reported  that  the  AGSS  did  not  fly  like  the  aircraft.  Instead,  they  cited  low  vibration 
level,  lack  of  gravity,  and  lack  of  seat-of-the-pants  feel. 


Physiological  Symptoms 

Self-reports  of  the  frequency  and  severity  of  nine  different  simulator  sickness  symptoms 
(visual  discomfort,  headaches,  double  vision,  blurred  vision,  disorientation,  eye  strain  or  fatigue, 
neck  strain,  stomach  discomfort,  and  nausea)  during  and  after  AGSS  sessions  were  obtained.  To 
determine  the  number  of  subjects  reporting  symptoms,  we  first  collapsed  across  all  of  the 
symptoms.  We  found  that  54%  of  the  survey  participants  reported  symptoms  sometimes  or  rarely 
and  46%  reported  never  having  symptoms  (both  during  and  after  AGSS  flights).  This  is  consistent 
with  the  literature  on  simulator  sickness,  where  the  percentage  of  individuals  reporting  symptoms 
across  a  selection  of  simulators  has  been  observed  between  10%  and  60%  (Kennedy,  Lilienthal, 
Berbaum,  Baltzley,  &  McCauley,  1989).  Figure  10  depicts  the  reported  incidence  of  simulator 
sickness  symptoms  during  AGSS  sessions. 
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Figure  10.  Reported  Incidence  of  Any  Simulator  Sickness  Symptoms  During  AGSS  Training. 
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Training  Capability 

Finally,  we  asked  participants  to  rate  from  1  (poor)  to  5  lexcentinnan  ti^  +^0:  • 
capability  of  the  AGSS  on  39  required  AG/S  skills.  For  this  assessment,  we  covered  a  broad 
of  skills,  including  ones  the  AGSS  was  not  explicitly  built  to  help  train.  Specifically  we  wanted 
to  assess,  (a)  whether  the  device  was  considered  capable  of  training  the  skills  it  wasbuilt  to  help 
(e.g.,  target  acquisition);  (b)  whether  there  were  skills  that  the  AGSS  was  not  explicitly 
designed  to  tram  (e.g.,  gun  malfunctions)  that  were  considered  “trainable”  in  the  AGSS-  and  let  to 
wha,  extent  these  skills  were  trainable  in  die  AGSS.  In  order  to  extract  L Mom!ftio7 i ll 

Sto^tLTteSTa*6  m“gnty,0f  ^  mtire  scale  “  our  analysis,  rather  than  collapsing  the 

probed  participants  for  training  capability  ratta^  for^toS^  WVG^ghTco^fctn0Thdr 

Table  13  lists  the  AG/S  skills  assessed  in  rank  order  of  their  mean  rating  nf  apcc  ^  •  • 
capability.  Their  associated  standard  deviations  are  presented  in  the  third  column  This  rank 
nng  simply  illustrate  which  skills  participants  feel  the  AGSS  is  best  suited  to  train  Although 
rank  ordering  alone  is  informative,  we  thought  it  would  be  worthwhile  to  group  these  skills  into 

categones  using  cluster  analysis,  as  previously  described.  ®  P  these  skills  mto 

Table  13.  Mean  Ratings  (Descending  Order)  and  Standard  Deviations  of  AGSS  Training 

Capability  for  39  AG/S  Skills. 


Gunner/Scanner  Skill 

Voice  Procedures 

4.36 

Leit,  Right,  I  ail  Call  Responsibilities 

4.18 

.60 

Lett,  Right,  Tail  Interactions 

4.09 

.54 

Taxi  and  Hover  Calls 

4.00 

.63 

Crew  Coordination/Overall 

4.00 

.63 
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Table  13.  Concluded. 


Threat  Breaks 

3.91 

.83 

Scanning 

3.91 

.94 

Tactical  Knowledge 

3.82 

.87 

Radar 

3.82 

.98  | 

Go-Arounds 

3.82 

.87 

AAA 

3.82 

.98 

Rules  of  Engagement 

3.82 

.87 

Defensive  Countermeasures 

3.80 

.63 

Approaches  and  Landings 

3.73 

1.01 

Gunner  Cockpit  Exchanges 

3.73 

1.10 

Ammo  Conservation 

3.73 

1.19 

Tactical  Approaches 

3.64 

1.03 

Terrain  Masking 

3.55 

.69 

Safety  and  Judgment 

3.45 

.95 

Terrain  Obstacle  Avoidance 

3.45 

.82 

Search  Operations 

3.36 

1.21 

Communication  System 

3.36 

1.03 

Aircrew  Briefing 

3.36 

1.36 

Remote  Operations 

3.27 

1.19 

Tactical  Mission  Navigation 

3.18 

1.38 

Radio  Discipline 

3.09 

1.38 

Use  Limitations 

3.09 

1.45 

Waypoint  Identification 

3.09 

1.04 

Target  Acquisition 

2.91 

1.04 

Knowledge  of  Directives 

2.64 

.81 

Passenger  Brief 

2.18 

1.54 

Resource  Management 

1.91 

.70 

Checklist  Procedures 

1.55 

.69 

NVG  Use  and  Limitations 

1.20 

.63 

Light  Signals 

1.18 

.40 

Emergency  Procedures 

1.18 

.40 

Gun  Malfunctions 

1.09 

.30 

NVG  Failure 

1.00 

.00 

Aircraft  Taping  and  Lighting 

1.00 

.00 

When  using  cluster  analysis  there  is  often  no  single  solution.  Rather  it  is  driven  by 
experimenter  interpretation  as  to  where  the  optimal  solution  is  in  terms  of  the  objects  clustered. 
Figure  11,  although  still  having  17  distinct  clusters,  was  selected  (forming  at  the  17th  step)  as  the 
best  grouping  of  the  skills  rated.  We  chose  this  step  primarily  because  of  the  four  large  clusters 
that  formed  (although  some  of  the  two-skill  clusters  are  also  informative).  These  four  large 
clusters  separate  the  skills  into  distinct  AG/S  skill  areas.  The  first  cluster  is  crew  coordination 
skills.  The  second  is  terminal  area  operation  skills.  The  third  is  tactical  skills.  And,  the  fourth  is 
“skills  the  AGSS  was  not  designed  to  train”  (requiring  actual  guns  or  NVGs  to  train).  This  is  not 
meant  to  imply  that  the  AGSS  was  designed  and  built  to  train  the  remaining  32  skills,  however  this 
category  reflects  the  skills  that  the  AGSS  was  never  meant  to  train. 
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Figure  11.  Cluster  Analysis  Output  of  39  AG/S  Skills. 

Several  interesting  points  arise  from  the  rank  orderings,  standard  deviations,  and  cluster 
analysis  combined.  First,  and  perhaps  most  important,  there  was  considerable  agreement  among 
the  raters  (as  evident  by  the  small  standard  deviations  listed  in  Table  13)  concerning  both  the 
highly  trainable  (first  three  large  clusters)  and  totally  untrainable  skills  (last  large  cluster)  in  the 
AGSS.  For  the  middle-range  skill  ratings  (3s),  there  was  more  variability.  Some  of  the  comments 
associated  with  these  skills  suggested  uncertainty,  not  based  on  the  device,  but  because 
participants  had  not  used  the  device  in  this  capacity  before.  Other  comments  suggested  that  some 
training  in  these  skill  areas  was  possible  with  a  little  instructor  imagination,  but  that  it  would  not 
be  as  complete  as  in  the  aircraft.  Combined,  these  factors  led  us  to  believe  that  there  may  be 
considerable  untapped  training  potential  of  the  AGSS  across  these  middle-range  skill  areas.  Table 
13  also  highlights  one  notable  discrepancy  between  one  of  the  key  skills  the  AGSS  was  explicitly 
built  to  train,  target  acquisition,  and  its  low  mean  training  capability  rating  by  participants.  Three 
specific  reasons  were  cited  for  participants’  low  ratings:  poor  visuals,  inability  to  judge  distances 
and  inaccurate  modeling  of  the  bullet  path. 
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SUMMARY  OF  RESULTS  AND  RECOMMENDATIONS 

All  participants  in  the  survey  were  quite  positive  about  the  training  potential  of  the  AGSS. 
Indeed,  the  most  frequent  positive  comment  across  the  board,  even  with  the  number  of  participant 
criticisms,  was  that  the  device  provided  unique  training  capabilities  unavailable  prior  to  the 
introduction  of  the  AGSS  into  the  58  TRSS.  With  this  said,  participants’  ratings  and  comments 
were  extremely  insightful  as  to  improvements  that  could  make  the  device  an  outstanding  training 
medium  for  AG  and  FE  training  curricula.  To  summarize,  the  central  findings,  both  positive  and 
negative,  associated  with  each  section  of  the  survey  are  presented  in  Table  14. 


Table  14.  Main  Findings  from  Each  Section  of  the  Survey:  Problems  and  Positives. 


Survey 

Section 

Main  Problems  Reported 

Positive  Aspects  Noted 

Helmet 

=>  CRT  failure  rate  and  discrepancies 
=>  Head-tracker:  slow  update  rate 
=>  Alignment  and  fitting  problems 
=>  Pads:  type  and  method  of  attachment  to  helmet 
=>  Cords:  too  short  for  requirements  of  tail  position 
and  they  sometimes  get  caught  during  training 

=>  Lightweight,  less  than  those  used  in 
the  aircraft 

Trainee 

Stations 

=>  Virtual  mapping  of  stations  and  guns,  a  few  noted 
problems 

=>  Mapping  of  outside  of  ownship 

IOS 

=>  Second  instructor  position  lacks  the  ability  to  talk 
directly  with  students  in  the  AGSS 
=>  Space  ball:  poor  modeling  of  own  ship 
aerodynamics,  very  difficult  to  use 
=>  Task  saturation:  monitoring  students,  running  IPS 
flying  moving  models,  loading  threats,  etc.  becomes 
quickly  overwhelming 

=>  Layout  of  screens  potentially  a  contributor  to  task 
saturation  and  instructor  inefficiency 

=>  Threats  on  ground  track  map  page 
=>  Similarities  with  other  58  TRSS 

IOSs  makes  it  easy  to  learn 

Weapons 

Simulation 

=>  Poor  modeling  of  bullet  path,  no  lead  and  lag 
=>  Insufficient  number  of  tracers  viewed  per  rounds 
fired 

Visuals 

=>  Alignment  procedures 

=>  Alignment  of  virtual  and  physical  environment 

=>  Poor  color  quality 

=>  Visual  continuity  (slow  update  rate) 

=>  Limited  FOV  for  daytime  operations 

=>  Explosions,  including  secondaries 

Physiological 

Symptoms 

=>  50%  of  participants  noted  the  occurrence  of 

simulator  sickness  symptoms  during  and  after  AGSS 
training 

=>  Eye  strain  and  visual  discomfort  were  reported  by  a 
significant  proportion  of  survey  participants 

=>  Reported  symptoms  did  not  disrupt 
training 

Training 

Capability 

=>  Visuals  make  it  difficult  to  train  certain  tasks 

=>  Outstanding  tool  for  training  voice 
procedures,  terminology,  and  crew 
coordination 

=>  Rated  training  capability  will  be 
higher  when  the  AGSS  is  integrated 
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Additionally,  throughout  this  report,  we  have  made  recommendations  for  improving  those 
AGSS  components  that  trainees  rated  unfavorably  and  which  were  frequently  criticized.  In  Table 
1 5,  we  summarize  the  main  problem  areas  and  our  recommendations  for  their  improvement.  The 
table  provides  gross  estimates  of  the  cost  and  training  '"payoff’  associated  with  each 
recommendation. 


Table  15.  Main  AGSS  Problem  Areas,  Recommendations  for  Improvement,  Associated  Cost 
Estimates,  and  Potential  Training  Payoff  Estimates. 


Problem  Area 

Recommendation 

Cost 

T  raining 
Payoff 

CRT  Failure  Rate 

=>  Invest  in  higher  quality  CRTs. 

HIGH 

HIGH 

CRT  Discrepancies 

=>  Purchase  new  CRTs  more  frequently. 

MEDIUM 

HIGH 

Slow  Head-tracker  Update  Rate 

=>  Invest  in  higher  quality,  more  recent 
technology. 

HIGH 

MEDIUM 

=>  Introduce  slower  NVG  scanning 
rates  early  on  in  AGSS  training. 

NONE 

HIGH 

Poor  Helmet  Alignment  and  ; 
Fitting  Procedures 

LOW 

Poor  Pads 

=>  Invest  in  higher  quality  pads 

LOW 

HIGH 

Length  of  Electronic  and 
Communication  Cables 

=>  Invest  in  longer  cables. 

MEDIUM 

MEDIUM 

=>  Provide  a  means  of  tethering  cables 
off  the  students’  shoulders. 

LOW 

MEDIUM 

Virtual  Mapping  of  Stations  (re: 
other  players  in  other  positions) 

=>  Map  players  in  all  positions. 

LOW* 

LOW 

=>  Do  not  map  any  players  in  other 
positions. 

LOW 

LOW 

Second  Instructor  Position’s 

Lack  of  Communication 
Capability 

,  =>  Provide  capability  to  communicate 
directly  with  students  in  the  AGSS. 

MEDIUM 

VERY  HIGH 

Problems  with  Space  Ball  for 
Ownship  Flying 

HIGH 

MEDIUM 

=>  Change  controls  to  a  joystick. 

HIGH 

HIGH 

NONE 

LOW 

High  Levels  of  Instructor 

Reported  Task  Saturation  at  IOS 

=>•  Operate  only  in  integrated  mode. 

HIGH 

MEDIUM 

=>  Limit  number  of  students  in  AGSS. 

MEDIUM 

VERY  LOW 

=>  Provide  two  instructors  for  training. 

MEDIUM 

HIGH 

=>  Change  layout  of  instructor  screens. 

MEDIUM 

HIGH 
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Table  15.  Concluded. 


Problem  Area 

Recommendation 

Cost 

Training 

Payoff 

Poor  Modeling  of  Bullet’s  Path 

=>  Change  modeling. 

HIGH 

VERY  HIGH 

=>  Provide  training  on  aircraft  versus 
AGSS  discrepancies. 

LOW 

MEDIUM 

Low  Number  of  Tracers  Visible 
Per  Rounds  Fired 

=>  Increase  number  of  tracers  per 
rounds  fired. 

LOW* 

MEDIUM 

. .  . . — - - 

=>  Provide  training  on  aircraft  versus 
AGSS  discrepancies  and  reasons. 

LOW 

MEDIUM 

■  LOW 

HIGH 

PoorObjectand  VisuaLScene 

HIGH 

Poor  GoIorQoality 

^ImprovecoJors.  . 

LOW 

Reduced  FOV  for  Daytime 
Operations 

=>  Change  size  of  windows  to 
accommodate  VR  helmet 

MEDIUM 

MEDIUM 

Poor  Visual  Quality 

=>  Improve  visual  systems. 

HIGH 

VERY  HIGH 

=>  Improve  other  systems  (e.g.,  CRTs) 

HIGH 

HIGH 

*NOTE-  Associated  costs  may  be  low,  but  due  to  current  limitations  in  the  IG,  these  suggestions  may  not  be 
possible. 


The  shaded  rows  in  the  table  indicate  areas  in  which  improvements  have  already  been 
made  to  the  AGSS  since  the  completion  of  the  survey.  Alignment  procedures  and  fitting 
procedures  have  been  greatly  improved,  and  the  current  level  of  expertise  is  high  among  the 
Lockheed  Martin  personnel  maintaining  and  working  with  the  AGSS.  However,  the  means  of 
transferring  this  knowledge  is  still  largely  through  demonstration  and  trial  and  error.  This 
suggests  a  need  to  impose  greater  documentation  and  dissemination  of  this  process,  such  as  by 
writing  out  step-by-step  instructions  for  distribution  to  all  AG/S  trainers.  The  modeling  of  the 
gun  and  its  pivot  point  has  also  been  improved  dramatically  since  the  survey,  and  observed 
responses  have  been  overwhelmingly  positive  in  terms  of  greater  student  acceptance  of  the  device. 
In  addition,  the  visual  system  and  various  aspects  of  the  data  bases  used  in  the  AGSS  are  in 
constant  flux.  As  a  consequence,  several  visual  features  of  the  AGSS  have  also  been  improved 
since  the  survey.  In  particular,  the  colors  and  the  detail  and  texturing  of  many  of  the  moving 
models  have  been  modified.  These  improvements  have  all  been  well  received  by  AGSS  students 
and  instructors. 


As  for  the  non-shaded  items  in  Table  10,  we  have  attempted  to  include  recommendations 
that  have  potentially  significant  training  payoffs.  The  cost  is  based  on  the  estimated  dollar  costs 
for  providing  the  hardware  or  software  modifications,  the  costs  of  extra  training  time  and/or 
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scheduling  of  additional  personnel  (e.g.,  operating  in  the  integrated  mode  or  using  an  additional 
AGSS  instructor). 


FUTURE  RESEARCH 

The  next  step  in  the  human  factors  analysis  of  the  AGSS  is  twofold:  (1)  user  acceptance 
and  (2)  training  effectiveness  (remember  Figure  2).  Both  of  these  areas  have  begun  to  be 
explored  by  the  Kirtland  AFB  research  team.  In  terms  of  user  acceptance,  the  58  TRSS  has 
created  a  series  of  questionnaires  for  instructors  and  students  to  determine  their  responses  to  the 
device  immediately  following  training  sessions.  The  questions  cover  a  wide  array  of  topics 
including,  human  factors  aspects  of  the  device,  training  provided,  and  device  fidelity. 

In  regard  to  training  effectiveness,  we  are  looking  at  device  impact  on  skill  acquisition  and 
performance  (largely  guided  by  the  skill  ratings  acquired  from  this  survey,  see  Table  12)  and 
AG/S  training  hours  in  the  aircraft.  Without  going  into  too  much  detail,  we  are:  (a)  observing 
training  for  five  scheduled  Mission  Qualification  AGSS  missions,  (b)  attaining  skill  ratings  of 
trainees  by  instructors  for  these  missions,  (c)  obtaining  objective  performance  pages  when 
possible  for  these  missions,  (d)  securing  ratings  of  skill  performance  from  the  flight  fine  on 
students  receiving  AGSS  training,  and  (e)  analyzing  AG/S  training  folder  data.  It  is  through  these 
primary  means  that  we  hope  to  determine  the  impact  of  the  AGSS  to  the  AG  training  system. 

Finally,  because  these  various  mechanisms  are  in  place,  we  can  also  address  the  training 
effects  of  system  upgrades  potentially  resulting  from  this  survey.  For  example,  the  AGSS  has 
currently  been  used  solely  as  a  stand-alone  trainer  because  of  delays  in  the  integration  process. 
We  will  have  several  months  of  data  on  a  significant  number  of  students  who  received  only  AGSS 
stand-alone  training.  As  soon  as  the  device  becomes  integrated,  we  will  be  able  to  compare  those 
students  skills  and  device  opinions  to  those  who  start  to  receive  both  stand-alone  and  integrated 
AGSS  training. 
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I 


HELMET  -  Did  you  experience  any  breakage. 


N  -  Never  R  -  Rare 

1 .  Visor? 

QdEKsKZV 

es  O-Often  A-Aiwavs 

1.  Poor  helmet  fit? 

NR  <?  n  a 

2.  Eye  Piece? 

N  R  S  O  A 

2.  Eye-piece  misalignment? 

N  R  Si  n  a 

3.  Head-tracker? 

N  R  S  O  A 

3.  Slippage? 

4.  Comm,  cable? 

N  R  S  0  A 

4.  Neck  fatique?  ’ 

5.  Elect,  cable? 

N  R  S  O  A 

5.  Hot  spots?  1 

6.  Other 

N  R  S  O  A 

6.  Chin  straps  chafinq? 

N  R  S  O  A 

7.  Chin  straps  chokinq? 

N  R  S  O  A 

8.  Unbalanced  weight? 

N  R  S  O  A 

9.  General  discomfort? 

N  R  S  O'  A 

iu.  utner  N  R  S  O  A 

CABLES 


1 .  Do  the  cables  unduly  tuq  at  the  back  of  your  head'? 

Y 

N 

2.  Do  the  cables  limit  the  amount  of  normal  movement  around  the  aircraft? 

Y 

N 

4.  uo  tne  cables  hinder  normal  head  movements  (e.q..  scanning  ability)? 

Y 

N 

4.  Do  the  cables  interfere  with  task  performance  (e.q.,  tarqet  acquisition)? 

5.  If  YES.  indicate  which  tasks- 

Y 

N 

HELMET  ADJUSTMENTS 


1 .  How  long  did  it  take  to  put  on  and  adjust  the  helmet  the  first  time? 

2.  How  long  did  it  usually  take  to  put  on  and  adjust  the  helmet? 

8.  unce  the  helmet  was  adjusted,  did  you  require  readjustments  durinq  the  train'mq 
time?  a 

Y 

N 

AGSS  TRAINEE  STATIONS 


1  -  Poor  2 -Min.  Reg. 


3  -Standard 


4- 


Outstandinq 


5  -  Exceptional 


TRAINEE  STATION 
1  -  Overall  station  set-up? 


Rating 


WEAPONS  SIMULATION 


MiniGun  .50  Cal 


1.  Gun  Recoil? 


2.  Gun 


set-up? 
Gun  feel? 


2.  Gun  Sounds? 


3.  Gun  Position? 


4,  Virtual  mapping  -  acft.  envmt? 


5.  Virtual  mapping  -  other  stations? 


4.  Gun  Weight/Feel? 


5.  Gun  Controls? 


Mobility  within  station? 
Crew  seats  and  harnesses? 


6.  Arrstream  Effects? 


_7_ 

8. 


7_ 

8. 


Tracer  Fidelity? 


Ballistic  Model? 


9.  Gun  Alignment? 

10.  Target  Acquisition? 


11.  Other: 


INSTRUCTOR  OPERATING  STATION  -  IOS 


1 .  Does  the  noise  level  of  the  surrounding  simulators  affect  your  ability  to  train 
at  the  AGSS  IOS?  7 

Y 

N 

2.  While  at  the  IOS,  does  your  inability  to  see  the  student  in  the  AGSS  affect 
your  ability  to  instruct? 

Y 

N 

3.  uo  you  Become  fully  task  saturated  at  the  AGSS  IOS? 

With  1,  2,  or  3  students?  (Circle  choice) 

Y 

N 

4'  s,mu,ator  bay  <e-9-  the  Ashing  red  light  of  the 

- MH-53J  WST)  affect  your  ability  to  train  at  the  AGSS  IPS? _ 

Y 

N 
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INSTRUCTOR  OPERATING  STATION  -  IOS  (Continued 


1  -  Poor  2  -  Min.  Req.  3  -  Standard 


FEATURES 


1.  instructor  screens? 


2.  Information  content? 


3.  Instructor  screens  layout? 


4.  Student  screens  information  content? 


5.  Student  screens  layout? 


6.  Station  layout  (position  of  screens)? _ 


7.  Space  ball?  _ 


8.  Other: 


INSTRUCTOR  &  STUDENT  INTERACTION  and  COMMUNICATION  SYSTEM _ 

1 .  Did  you  have  any  problems  hearing/understanding  the  instructor  while  in  the  AGSS? 


2.  Did  you  have  any  problems  hearing/understanding  students  while  at  the  AGSS  IPS? 


3.  Did  you  have  any  problems  hearing  or  understanding  other  crewmembers? _ 


4.  Was  the  instructor  able  to  provide  timely  feedback  while  you  were  in  the  AGSS? 


roblems  with  communications  in  the  heimet/device? 


6.  Was  the  sound  quality  and  background  noise  comparable  to  the  aircraft? _ 


-  If  YES.  from  a  training  perspective,  is  this  good? _ _ 


-  If  NO,  from  a  training  perspective,  is  this  good? 


VISUAL  SYSTEM 


N  -  Never 


1 .  Visual  Discomfort? 


2.  Headache? 


3.  Double  Vision? 


4.  Blurred  Vision? 


5.  Disorientation? 


6.  Eve  Strain/Fatigue? 


7.  Neck  Strain? 


8.  Stomach  Discomfort? 


9.  Nausea? 


S  -  Sometimes  O  -  Often  A  -  Alw 


While  Flying  in 
AGSS? 


N 

R 

S 

o 

A 

N 

R 

s 

o 

A 

N 

R 

s 

o 

A 

N 

R 

s 

o 

A 

N 

R 

s 

o 

A 

N 

R 

s 

o 

A 

N 

R 

s 

o 

A 

NRSOAINRSOA 


NRSOAINRSOA 


VISUAL  SYSTEM  -  DAY  and  NIGHT  OPERATIONS 


1  1  -  Poor  2-Min.  Req.  3  -  Standard  4- Outstanding  5  -  Exceptional 

FEATURE 

Rating 

Representation  of  Specific  Objects 

Rating 

1 .  Visual  resolution  (object  detail)? 

1.  747  aircraft? 

2.  Visual  scene  clarity? 

2.  Airborne  helicopter? 

3.  Visual  scene  completeness? 

3.  Jeeps? 

4.  Visual  scene  continuity? 

4.  Trucks? 

5.  Meaningful  depth? 

5.  SAM  sites? 

6.  Image  stability? 

6.  Trees? 

7.  Image  clarity? 

7.  Roads? 

8.  Image  alignment? 

8.  Runways? 

9.  Image  color? 

9.  Explosions? 

10.  Visual  system  brightness? 

10.  Other? 

1 1 .  Visual  system  contrast? 

12.  Object  detectability? 

13.  Distance  Perception? 

14.  Other? 

15.  Overall  Visual  System  assessment? 

VISUAL  SYSTEM  -  Field-of-View 


1 .  Did  you  achieve  a  full  FOV? 

Y  |  N 

-  If  NO  what  items  were  you  unable  to  see?  (List  here) 

2.  Did  your  FOV  change  when  you  moved  your  head  Vertically? 

Y 

N 

3.  Did  your  FOV  change  when  you  moved  your  head  Horizontally? 

..  Y  : 

N  -3'. 

4.  Did  your  FOV  change  when  you  moved  your  head  Forward? 

Y 

N  ■ 

5.  Did  your  FOV  change  when  you  moved  your  head  Backward? 

Y  \'Y3 

W 

6.  Were  you  able  to  execute  a  normal  scan? 

Y 

'  •  N  ' 

DEVICE  CAPABILITY  FOR  TRAINING  CERTAIN  SKILLS 


1  -  Poor  2 -Mm.  Req.  3  -  Standard 


Rating 


Day  Night 


SKILL 


1 .  Voice  Procedures? 


2.  Checklist  Procedures? 


3.  Gunner  Cockpit  Exchanges? 


4.  Left/Right/Tail  call  resp.? 


5.  Left/Right/Tail  interactions? 


6.  Radio  Discipline? 


7.  Aircrew  briefing? 


8.  Passenger  briefing? 


9.  Other: 


10.  Crew  Coordination? 


11.  Safety  and  judgment? 


12.  Resource  management? 


13.  Knowledge  of  directives? 


14.  Communication  system? 


15.  Taxi  and  hover  calls? 


16.  Approaches  and  landings? 


17.  Go-arounds? 


18.  Scanning? 


19.  Search  operations? 


20.  Remote  operations? 


21.  Tactical  approaches? 


MISCELLANEOUS 


1  Did  the  AGSS  fly  like  the  aircraft? 


2.  Did  you  ever  feel  disoriented  in  the  AGSS? 

If  YES,  under  what  circumstance(s)?  ~ 


_ a.  During  Target  Acquisition? 


_ b.  During  TF/TA? 


_ c.  During  Takeoff? 


_ d.  During  Landings? 


_ e.  During  Crashes? 


_ f-  During  System  Reset? 


_ g.  When  Instructor  is  flying  with  the  space  ball? _ 

h.  When  flying  in  Left  Station?  “  " 

i.  When  flying  in  Right  Station?  -  "  ~ 

j.  When  flying  in  Tail  Station?  ' 

3.  If  you  ever  felt  disoriented,  did  a  particular  system  contribute  to  this  perception? 

4.  Were  you  satisfied  with  the  motion  system?  ~ 


SKILL 


Tactical  knowledge? 


Rules  of  Engagement? 


Terrain/Obstacle  avoid.? 


Waypoint  identification? 


Light  signals? 


Target  acquisition? 


Ammo  conserve? 


Use  limitations? 


Emergency  procedures? 


Gun  malfunctions? 


NVG  failure? 


Aircraft  taping  and  lighting? 


NVG  use  and  limitations? 


Radar? 


AAA  Threats? 


Terrain  masking? 


Threat  breaks? 


Countermeasures? 


Overall  Training  Capability? 
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APPENDIX  B 


jdM’JgaaHPilHSiS 


Component 


Visor 


Eve  Piece  Combiner 


Head-tracker 


Communication  Cable 


Electronics  Cable 


CRTs 


Helmet  Fit 


ent 


e 


Neck  Fatigue 


HotS 


Chin  S 


Unbalanced  Weight  Dist. 


General  Comfort 


Helmet  Pads 


Gun  Set  U 


Gun  Feel 


VR  Aircraft  Interior 


VR  Other  Stations 


Mobility  within  Station 


Crew  Seats 


Crew  Harnesses 


Overall 


Participant  Ratings:  Raw  Data 

Rating,  frequen 


Instructor  Screen  Content 


Instructor  Screen  Layout 


Student  Screen  Content 


Student  Screen  Layout 


Second  Instructor  Position 


Ground  Track 


Gunner  Performance  Page 


Crew  Performance  Page 


Threat  Status  Page 


Flare  Monitoring  Page 


■  1  ■ ; 

1  2 

3 

0 

0 

2 

0 

1 

2 

11 

0 

0 

0 

0 

5 

0 

1 

2 

0 

0 

3 

0 

2 

3 

1 

0 

3 

0 

0 

0 

0 

0 

3 

0 

0 

1 

1 

1 

2 

0 

2 

0 

0 

0 

1 

0 

1 

0 

0 

1 

2 

1 

1 

0 

-2  ■  - 

m  ■ .  ■ 

0 

1 

6 

0 

0 

0 

0 

3 

1 

0 

1 

5 

0 

1 

5 

0 

1 

2 

1 

0 

3 

0 

1 

3 

illlSll 

111131111 

SSI 

0 

l 

2 

0 

0 

2 

0 

i 

2 

0 

l 

3 

0 

0 

3 

2 

0 

1 

5 

3 

1 

1 

3 

2 

0 

0 

5 

0 

0 

2 

0 

0 

1 

0 

0 

4 

0 

0 

5 

0 

0 

2 
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Participant  Ratings:  Raw  Data 


Component 


Rating,  frequency 


Gun  Recoil _ 

Gun  Sounds _ 

53  Gun  Position 
60  Gun  Position 
Gun  Weight  and  Feel 

Gun  Controls _ 

Airstream  Effects 

Tracers _ 

Ballistic  Model 
Gun  Alignment 
Gun  Pivot  Point 
Target  Acquisition 
Overall  Rating _ 


Visual  Resolution:  Object  Pet. 

747 _ 

Hind-D _ 

Jeep _ 

Truck _ 

SAMs _ 

Trees _ 

Roads _ 

Runways _ 

Explosions _ 

Field  of  View 

Brightness _ 

Contrast _ 

Visual  Resolution:  Scene  Det. 

Object  Detectability _ 

Distance  Perception _ 

Scene  Clarity 
Scene  Completeness 

Scene  Continuity _ 

Scene  Depth 

Image  Stability _ 

Image  Clarity _ 

Alignment 
Image  Color. 

Overall  Rating 


Participant  Ratings:  Raw  Data 


Component 


Rating,  frequency 


Voice  Procedures 


Checklist  Procedures 


Gunner-Cockpit  Exchanges  _ 


Left,  Right,  Tail  Call  Resp. 


Left,  Right,  Tail  Interactions 


Radio  Discipline  _ 


Aircrew  Briefings 


Passenger  Briefings  _ 


Crew  Coordination  Overall 


Safety  and  Judgment  _ 


Resource  Management 


Knowledge  of  Directives 


Communication  Systems 


Taxi/Hover  Clearance  Calls 


Approach/Landing  Calls 


Go-Arounds 


Scanning  _ 


Search  Operations 


Remote  Operations 


Tactical  Approaches 


Tactical  Knowledge 


Radar/IR 


AAA 


Terrain  Masking 


Threat  Breaks/Evas.  Maneuvers 


Countermeasures 


Rules  of  Engagement 


Tactical  Navigation 


Terrain/Obstacle  Avoidance 


Waypoint  Identification 


Light  Signals 


Emergency  Procedures  _ 


Target  Acquisition _ 


Ammo  Conservation 


Use  Limitations 


Gun  Malfunctions 


NVG  Failure 


Aircraft  Taping  and  Lightmg 


NVG  Use  and  Limitations 
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